Building

Started building at 2021/12/08 10:39:07
Calculating base
Updating mirror
Basing run on 7.1.0-1863 0dcc308d2
Updating tree for run 08.12.2021-10.39
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f1a81ab, changes since last good build: 
 f1a81abb Merge remote-tracking branch 'couchbase/unstable' into HEAD
 235e3e2a MB-49356 Part 3 (7.1.0 1745): Runtime: Don't overwrite newer cache entry
 08499e0f MB-49838 Revert back to using collection seqnos for session consistent scans
 5f8d5ea4 Merge remote-tracking branch 'couchbase/unstable' into HEAD
 dd88c97c MB-49356 Part 2 (7.1.0 1745): Boot: don't overwrite newer cache entries
kv_engine is at 3b09efd, changes since last good build: 
 3b09efd5f MB-49804: [Nexus] Prune stale expiry callbacks [2/2]
 fc356c4fd MB-49804: [Nexus] Store seqno of expiry callbacks [1/2]
 67114143f MB-47604: Add collectionsPurged compaction stat [1/3]
 ceafb8382 Make collectionsItemsPurged update unconditional
 edd4083de MB-47687: Disable collection drop during implicit compaction
 b36b5316c MB-44697: Move collection logging from vbucket to bucket
 582455737 [Magma] Print CompactKVStore status on failure
 07f7eff68 MB-47687: Prevent drop of rollbackable tombstones in magma
 cfd930b3b MB-45185: [Magma] Decrement coll size by full value at deletion [3/3]
 5bdf3ba36 MB-49839: Return early in CouchKVStore::getAllKeys
 deea6e96a Clang-tidy & IWYU: bgfetcher.(h/cc)
 b054c034e Remove unused variable from MagmaKVStore::rollback
 7e1018882 BucketManager::forEach: remove unused arg
 c567c6eb3 MB-47687: Add highSeqno to magma UserStats
 8041faf88 MB-45185: Pass individual sizes to remove rather than delta [2/3]
 799790d8c MB-45185: Pass CompactionCallbacks to low level flush accounting [1/3]
 3104f01ac MB-49802: [Nexus] Add manifestUid/readyForCommit checks before commit
 60b579a3a MB-49525: Change defragmenter minimum sleep from 0.0 to 0.6
 f2fcc5e5d MB-49847: Update tests to cover minor DCP OSO issue
 806c8485f IWUY: stat_checker.(h/cc)
ns_server is at fcadff5, changes since last good build: 
 fcadff5e7 Make leader_quorum_nodes_manager less noisy.
 ff16f5221 Move notifying processes about node name changes to dist_manager.
 44adb94f5 MB-49449: Make plasma the default index storage mode option
 374a4c3e2 MB-48003: Don't try to load PKCS12 file on joinee
 5e70c784a MB-49166: Don't allow modification of allowedHosts if...
 4ed657e64 Introduce addr validation fun in ns_cluster:maybe_rename
 8a79a0f70 MB-49166: Check allowedHosts predicate when adding nodes
 921ac75da Refactor check_add_possible to avoid nested cases
 ce90fc4af MB-49166: Add allowedHosts param to /clusterInit
 b6531d345 MB-49166: Allow getting of allowedHosts from non-localhost
 d832aa7db MB-49166: Add /security/settings/allowedHosts
 e9278bf1d MB-48003: Disable empty passwords for pkcs12 because...
 0c5e201e9 MB-48003: Add support for PKCS12
 36151eaa2 MB-49691: Add 3DES ciphers to low_security_ciphers list.
 38034762e MB-49691: Add RSA-PSK-RC4-SHA to low_security_ciphers list.
 4d4aac866 Add scripts/master_events_2_gtrace
 cb861afab MB-49885 Correct wording on alerts page
 bd3e8d2ce MB-49314 Prevent GET /diag/masterEvents?o=1 failure post bucket creation
 6f96ff41f MB-49184: allow user without xdcr.remote_clusters!read to open XDCR
 84cb01cc6 MB-49812: use catchErrors to handle postIndexSettings just_validate
 6d7a5f386 MB-49812: use catchErrors to handle postPoolsDefault just_validate
 c5422c2a8 MB-49812: handle postAutoReprovision/postAutoFailover just_validate 400
 ed6ee2e74 MB-49812: handle postSettingsRetryRebalance just_validate 400
 caf938e0d MB-49812: add comment about why we handle postCreateReplication success
 fab2cc81e MB-49812: handle postSettingsWebHttp just_validate 400
 59eb00280 Return 400 when just_validate=1 and validation fails
 e3db0a903 MB-48738: install missed worker-*.js files
 33b5af5ac MB-48738: rm -rf node_modules folder after npm run rebuild
 99d62e323 MB-48738: remove previous js build before running new one
 31a3c822b MB-48738: generate SourceMapLinked files - '*.js.map'
 91d37e301 MB-48738: keep web_modules js files unminified
 d632fc8bd MB-48738: exclude node_modules from install
 bea4a444c MB-48738: do not install html dev files
 fdcab7c65 MB-49563: Change auto-failover limit label in Neo
 5cde18981 MB-49210: Replace save bucket logic with the AngularJS one
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at a1f49af, changes since last good build: 
 a1f49af9c to use http/port for community edition
 bc2d8f989 free tier limits tests
 0095f8584 CBQE-7355: Fix add_node for upgrade tests with <6.5
 d6e26b762 CBQE-7374: Fix TLS tests in backup_restore
 e84a24d3d CBQE-7358: Add tests for resuming restores in CBBM
 ea6e42e30 add a recursive test and a try_catch test.
 8e6c5eaf3 CBQE-7377 enable checking rza not in CE and turn on after migrating from CE to EE.
 23dfe0c55 Multiple CA tests
 b73248e8c N1ql UDF tests.
 ae5405235 CBQE-7362: Update system event log parsing for new format
 c225b6a6d CBQE-7350: Fix GCP teardown with tests in parallel
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 80b89f60] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable f6a1f141] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable f8965fe7] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 543525c9] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable b263a976] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable cd319a86] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 08aff58] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/08 10:55:17
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/08 10:56:46 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57068->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:57068->127.0.0.1:9000: read: connection reset by peer
2021/12/08 10:56:56 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 10:57:01 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/08 10:57:01 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 10:57:06 Removing node(s): [] from the cluster
2021/12/08 10:57:11 Rebalance progress: 0
2021/12/08 10:57:17 Rebalance progress: 0.9584780092592594
2021/12/08 10:57:21 Rebalance progress: 5.750868055555556
2021/12/08 10:57:27 Rebalance progress: 9.25925925925926
2021/12/08 10:57:31 Rebalance progress: 13.70804398148148
2021/12/08 10:57:36 Rebalance progress: 18.51851851851852
2021/12/08 10:57:41 Rebalance progress: 21.75564236111111
2021/12/08 10:57:47 Rebalance progress: 26.56611689814815
2021/12/08 10:57:51 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/08 10:57:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:58:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:58:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:58:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:58:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:59:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:59:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:59:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 10:59:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:00:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:00:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:00:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:00:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:01:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:01:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:01:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:01:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/08 11:02:12 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:02:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:02:18 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 11:02:23 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 11:02:28 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 11:02:33 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/08 11:02:33 App: TestAdvancedGetOps got deployed
2021/12/08 11:02:33 Starting bucket ops to source bucket
2021/12/08 11:02:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:02:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:02:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:02:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:02:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:02:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:02:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 11:02:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 11:02:59 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 11:03:04 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/08 11:03:04 App: TestAdvancedGetOps got un-deployed
2021/12/08 11:03:04 No eventing-con process running
2021/12/08 11:03:04 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/08 11:03:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:03:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:03:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:03:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:03:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (330.14s)
=== RUN   TestAdvancedInsertOps
2021/12/08 11:03:27 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:03:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:03:33 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:03:38 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:03:43 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:03:48 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:03:48 App: TestAdvancedInsertOps got deployed
2021/12/08 11:03:48 Starting bucket ops to source bucket
2021/12/08 11:03:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 11:03:59 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 11:03:59 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 11:03:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:03:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:03:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:04:04 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:04:09 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:04:14 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:04:19 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:04:19 App: TestAdvancedInsertOps got un-deployed
2021/12/08 11:04:19 No eventing-con process running
2021/12/08 11:04:19 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/08 11:04:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:04:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 11:04:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 11:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:04:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:04:48 Waiting for dst bucket item count to get to: 0 curr count: 683
2021/12/08 11:04:53 Waiting for dst bucket item count to get to: 0 curr count: 171
2021/12/08 11:04:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:04:53 Testing insert operation on source bucket
2021/12/08 11:04:53 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:04:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:04:59 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:05:04 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:05:09 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:05:14 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/08 11:05:14 App: TestAdvancedInsertOps got deployed
2021/12/08 11:05:14 Starting bucket ops to source bucket
2021/12/08 11:05:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 11:05:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/08 11:05:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 11:05:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:05:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:05:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:05:30 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:05:35 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:05:40 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:05:45 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/08 11:05:45 App: TestAdvancedInsertOps got un-deployed
2021/12/08 11:05:45 No eventing-con process running
2021/12/08 11:05:45 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/08 11:05:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:05:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 11:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:05:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.46s)
=== RUN   TestAdvancedUpsertOps
2021/12/08 11:06:04 Starting bucket ops to source bucket
2021/12/08 11:06:04 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:06:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:06:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:06:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:06:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:06:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:06:25 App: TestAdvancedUpsertOps got deployed
2021/12/08 11:06:25 Starting bucket ops to source bucket
2021/12/08 11:06:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 11:06:37 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 11:06:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:06:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:06:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:06:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:06:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:06:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:06:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:06:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:06:57 App: TestAdvancedUpsertOps got un-deployed
2021/12/08 11:06:57 No eventing-con process running
2021/12/08 11:06:57 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/08 11:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:07:08 Waiting for dst bucket item count to get to: 0 curr count: 835
2021/12/08 11:07:13 Waiting for dst bucket item count to get to: 0 curr count: 323
2021/12/08 11:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:07:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:07:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:07:20 Testing upsert operation on source bucket
2021/12/08 11:07:20 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:07:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:07:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:07:31 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:07:36 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:07:41 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/08 11:07:41 App: TestAdvancedUpsertOps got deployed
2021/12/08 11:07:41 Starting bucket ops to source bucket
2021/12/08 11:07:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:07:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:07:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:07:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:07:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:07:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:07:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:08:03 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/08 11:08:03 App: TestAdvancedUpsertOps got un-deployed
2021/12/08 11:08:03 No eventing-con process running
2021/12/08 11:08:03 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/08 11:08:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:08:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.99s)
=== RUN   TestAdvancedReplaceOps
2021/12/08 11:08:17 Starting bucket ops to source bucket
2021/12/08 11:08:18 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:08:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:08:24 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:08:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:08:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:08:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:08:39 App: TestAdvancedReplaceOps got deployed
2021/12/08 11:08:39 Starting bucket ops to source bucket
2021/12/08 11:08:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 11:08:50 Waiting for dst bucket item count to get to: 0 curr count: 850
2021/12/08 11:08:55 Waiting for dst bucket item count to get to: 0 curr count: 850
2021/12/08 11:08:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:08:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:08:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:08:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:09:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:09:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:09:10 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:09:15 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:09:15 App: TestAdvancedReplaceOps got un-deployed
2021/12/08 11:09:15 No eventing-con process running
2021/12/08 11:09:15 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/08 11:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:09:27 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/08 11:09:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:09:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:09:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:09:34 Testing upsert operation on source bucket
2021/12/08 11:09:34 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:09:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:09:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:09:45 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:09:50 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:09:55 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/08 11:09:55 App: TestAdvancedReplaceOps got deployed
2021/12/08 11:09:55 Starting bucket ops to source bucket
2021/12/08 11:09:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:09:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:09:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:09:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:10:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:10:06 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:10:11 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:10:16 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/08 11:10:16 App: TestAdvancedReplaceOps got un-deployed
2021/12/08 11:10:16 No eventing-con process running
2021/12/08 11:10:16 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/08 11:10:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:10:28 Waiting for dst bucket item count to get to: 0 curr count: 150
2021/12/08 11:10:33 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/08 11:10:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.81s)
=== RUN   TestAdvancedDeleteOps
2021/12/08 11:10:39 Starting bucket ops to source bucket
2021/12/08 11:10:40 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:10:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:10:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:10:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:10:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:11:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:11:01 App: TestAdvancedDeleteOps got deployed
2021/12/08 11:11:01 Starting bucket ops to source bucket
2021/12/08 11:11:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:11:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:11:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:11:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:11:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:11:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:11:16 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:11:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:11:26 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:11:31 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:11:31 App: TestAdvancedDeleteOps got un-deployed
2021/12/08 11:11:31 No eventing-con process running
2021/12/08 11:11:31 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/08 11:11:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:11:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:11:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:11:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:11:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:11:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:11:55 Testing delete operation on source bucket
2021/12/08 11:11:55 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:11:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/08 11:12:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:12:06 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:12:11 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:12:16 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/08 11:12:16 App: TestAdvancedDeleteOps got deployed
2021/12/08 11:12:16 Starting bucket ops to source bucket
2021/12/08 11:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:12:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:12:16 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:12:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:12:21 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:12:26 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:12:31 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:12:36 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/08 11:12:36 App: TestAdvancedDeleteOps got un-deployed
2021/12/08 11:12:36 No eventing-con process running
2021/12/08 11:12:36 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/08 11:12:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:12:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.77s)
=== RUN   TestEnoentAdvancedGet
2021/12/08 11:12:50 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:12:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:12:56 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:13:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:13:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:13:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:13:11 App: TestEnoentAdvancedGet got deployed
2021/12/08 11:13:11 Starting bucket ops to source bucket
2021/12/08 11:13:17 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:13:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:13:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:13:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:13:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:13:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:13:27 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:13:32 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:13:37 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:13:42 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:13:42 App: TestEnoentAdvancedGet got un-deployed
2021/12/08 11:13:42 No eventing-con process running
2021/12/08 11:13:42 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/08 11:13:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:13:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:13:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:13:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:14:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:14:10 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/08 11:14:15 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/08 11:14:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:14:15 Testing get enoent operation on source bucket
2021/12/08 11:14:15 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:14:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:14:21 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:14:26 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:14:31 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:14:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/08 11:14:36 App: TestEnoentAdvancedGet got deployed
2021/12/08 11:14:36 Starting bucket ops to source bucket
2021/12/08 11:14:42 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 11:14:42 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 11:14:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:14:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:14:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:14:47 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:14:52 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:14:57 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:15:02 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/08 11:15:02 App: TestEnoentAdvancedGet got un-deployed
2021/12/08 11:15:02 No eventing-con process running
2021/12/08 11:15:02 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/08 11:15:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:15:13 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 11:15:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.03s)
=== RUN   TestEnoentAdvancedReplace
2021/12/08 11:15:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:15:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:15:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:15:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:15:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:15:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:15:46 App: TestEnoentAdvancedReplace got deployed
2021/12/08 11:15:46 Starting bucket ops to source bucket
2021/12/08 11:15:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:15:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:15:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:15:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:15:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:15:57 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:16:02 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:16:07 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:16:12 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:16:12 App: TestEnoentAdvancedReplace got un-deployed
2021/12/08 11:16:12 No eventing-con process running
2021/12/08 11:16:12 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/08 11:16:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:16:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:16:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:16:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:16:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:16:40 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/08 11:16:45 Waiting for dst bucket item count to get to: 0 curr count: 71
2021/12/08 11:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:16:45 Testing get enoent operation on source bucket
2021/12/08 11:16:45 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:16:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:16:51 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:16:56 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:17:02 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:17:07 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/08 11:17:07 App: TestEnoentAdvancedReplace got deployed
2021/12/08 11:17:07 Starting bucket ops to source bucket
2021/12/08 11:17:12 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 11:17:12 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 11:17:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:17:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:17:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:17:17 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:17:22 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:17:27 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:17:32 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/08 11:17:32 App: TestEnoentAdvancedReplace got un-deployed
2021/12/08 11:17:32 No eventing-con process running
2021/12/08 11:17:32 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/08 11:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:17:44 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 11:17:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:17:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:17:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:17:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.02s)
=== RUN   TestEnoentAdvancedDelete
2021/12/08 11:17:55 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:17:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 11:18:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:18:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:18:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:18:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:18:16 App: TestEnoentAdvancedDelete got deployed
2021/12/08 11:18:16 Starting bucket ops to source bucket
2021/12/08 11:18:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:18:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:18:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:18:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:18:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:18:27 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:18:32 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:18:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:18:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:18:42 App: TestEnoentAdvancedDelete got un-deployed
2021/12/08 11:18:42 No eventing-con process running
2021/12/08 11:18:42 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/08 11:18:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:18:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:18:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:18:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:19:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:19:10 Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/08 11:19:15 Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/08 11:19:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:19:15 Testing get enoent operation on source bucket
2021/12/08 11:19:15 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:19:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 11:19:21 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:19:26 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:19:31 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:19:36 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/08 11:19:36 App: TestEnoentAdvancedDelete got deployed
2021/12/08 11:19:36 Starting bucket ops to source bucket
2021/12/08 11:19:42 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/08 11:19:42 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/08 11:19:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:19:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:19:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:19:47 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:19:52 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:19:57 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:20:02 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/08 11:20:02 App: TestEnoentAdvancedDelete got un-deployed
2021/12/08 11:20:02 No eventing-con process running
2021/12/08 11:20:02 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/08 11:20:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:20:13 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/08 11:20:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:20:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:20:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.92s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/08 11:20:25 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:20:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/08 11:20:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:20:36 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:20:41 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:20:46 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:20:46 App: TestAdvancedInsertKeyExist got deployed
2021/12/08 11:20:46 Starting bucket ops to source bucket
2021/12/08 11:20:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:20:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:20:47 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:20:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:20:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:20:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:21:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:21:07 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:21:07 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/08 11:21:07 No eventing-con process running
2021/12/08 11:21:07 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/08 11:21:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:21:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:21:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:21:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:21:21 Pumping document in the destination bucket
2021/12/08 11:21:21 Starting bucket ops to source bucket
2021/12/08 11:21:21 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:21:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/08 11:21:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:21:32 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:21:37 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:21:42 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/08 11:21:42 App: TestAdvancedInsertKeyExist got deployed
2021/12/08 11:21:42 Starting bucket ops to source bucket
2021/12/08 11:21:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:21:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:21:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:21:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:21:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:21:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:21:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:22:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:22:07 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:22:12 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/08 11:22:12 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/08 11:22:12 No eventing-con process running
2021/12/08 11:22:12 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/08 11:22:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:22:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:22:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:22:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:22:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:22:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.68s)
=== RUN   TestExipryGet
2021/12/08 11:22:36 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:22:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:22:42 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:22:47 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:22:52 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:22:57 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:22:57 App: TestExipryGet got deployed
2021/12/08 11:22:57 Starting bucket ops to source bucket
2021/12/08 11:23:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:23:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:23:52 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:23:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:23:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:23:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:23:58 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:24:03 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:24:08 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:24:13 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:24:13 App: TestExipryGet got un-deployed
2021/12/08 11:24:13 No eventing-con process running
2021/12/08 11:24:13 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/08 11:24:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:24:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:24:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:24:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:24:27 Pumping document in the destination bucket for expiry get
2021/12/08 11:24:27 Starting bucket ops to source bucket
2021/12/08 11:24:27 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:24:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:24:33 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:24:38 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:24:43 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:24:48 Waiting for app: TestExipryGet to get deployed
2021/12/08 11:24:48 App: TestExipryGet got deployed
2021/12/08 11:24:48 Starting bucket ops to source bucket
2021/12/08 11:25:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:25:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:25:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:25:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:25:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:25:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:25:48 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:25:53 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:25:58 Waiting for app: TestExipryGet to get un-deployed
2021/12/08 11:25:58 App: TestExipryGet got un-deployed
2021/12/08 11:25:58 No eventing-con process running
2021/12/08 11:25:58 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/08 11:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:26:10 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/08 11:26:15 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/08 11:26:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:26:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (226.15s)
=== RUN   TestExipryInsert
2021/12/08 11:26:22 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:26:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/08 11:26:28 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:26:33 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:26:38 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:26:43 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:26:43 App: TestExipryInsert got deployed
2021/12/08 11:26:43 Starting bucket ops to source bucket
2021/12/08 11:27:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:27:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:27:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:27:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:27:28 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:27:33 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:27:39 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:27:44 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:27:44 App: TestExipryInsert got un-deployed
2021/12/08 11:27:44 No eventing-con process running
2021/12/08 11:27:44 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/08 11:27:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:27:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:27:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:27:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:27:58 Testing on the destination bucket for expiry insert
2021/12/08 11:27:58 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:27:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/08 11:28:04 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:28:09 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:28:14 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:28:19 Waiting for app: TestExipryInsert to get deployed
2021/12/08 11:28:19 App: TestExipryInsert got deployed
2021/12/08 11:28:19 Starting bucket ops to source bucket
2021/12/08 11:29:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:29:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:29:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:29:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:29:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:29:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:29:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:29:24 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:29:29 Waiting for app: TestExipryInsert to get un-deployed
2021/12/08 11:29:29 App: TestExipryInsert got un-deployed
2021/12/08 11:29:29 No eventing-con process running
2021/12/08 11:29:29 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/08 11:29:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:29:41 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/08 11:29:46 Waiting for dst bucket item count to get to: 0 curr count: 98
2021/12/08 11:29:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:29:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.58s)
=== RUN   TestExipryReplace
2021/12/08 11:29:53 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:29:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:29:59 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:30:04 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:30:09 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:30:14 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:30:14 App: TestExipryReplace got deployed
2021/12/08 11:30:14 Starting bucket ops to source bucket
2021/12/08 11:30:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:31:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:31:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:31:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:31:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:31:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:31:09 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:31:14 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:31:19 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:31:24 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:31:24 App: TestExipryReplace got un-deployed
2021/12/08 11:31:24 No eventing-con process running
2021/12/08 11:31:24 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/08 11:31:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:31:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:31:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:31:38 Testing on the destination bucket for expiry insert
2021/12/08 11:31:38 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:31:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:31:44 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:31:49 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:31:54 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:31:59 Waiting for app: TestExipryReplace to get deployed
2021/12/08 11:31:59 App: TestExipryReplace got deployed
2021/12/08 11:31:59 Starting bucket ops to source bucket
2021/12/08 11:32:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:32:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:32:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:32:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:32:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:32:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:33:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:33:05 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:33:10 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:33:15 Waiting for app: TestExipryReplace to get un-deployed
2021/12/08 11:33:15 App: TestExipryReplace got un-deployed
2021/12/08 11:33:15 No eventing-con process running
2021/12/08 11:33:15 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/08 11:33:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:33:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:33:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:33:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:33:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.57s)
=== RUN   TestExipryUpsert
2021/12/08 11:33:33 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:33:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/08 11:33:39 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:33:44 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:33:49 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:33:54 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:33:54 App: TestExipryUpsert got deployed
2021/12/08 11:33:54 Starting bucket ops to source bucket
2021/12/08 11:34:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:34:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:34:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:34:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:34:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:34:50 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:34:55 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:35:00 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:35:05 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:35:05 App: TestExipryUpsert got un-deployed
2021/12/08 11:35:05 No eventing-con process running
2021/12/08 11:35:05 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/08 11:35:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:35:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:35:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:35:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:35:19 Testing on the destination bucket for expiry insert
2021/12/08 11:35:19 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:35:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/08 11:35:25 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:35:30 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:35:35 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:35:40 Waiting for app: TestExipryUpsert to get deployed
2021/12/08 11:35:40 App: TestExipryUpsert got deployed
2021/12/08 11:35:40 Starting bucket ops to source bucket
2021/12/08 11:36:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:36:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:36:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:36:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:36:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:36:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:36:35 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:36:40 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:36:45 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:36:50 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/08 11:36:50 App: TestExipryUpsert got un-deployed
2021/12/08 11:36:50 No eventing-con process running
2021/12/08 11:36:50 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/08 11:36:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:37:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:37:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:37:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:37:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:37:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.71s)
=== RUN   TestCasReplace
2021/12/08 11:37:14 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:37:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:37:20 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:37:25 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:37:30 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:37:35 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:37:35 App: TestCasReplace got deployed
2021/12/08 11:37:35 Starting bucket ops to source bucket
2021/12/08 11:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:37:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:37:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:37:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:37:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:37:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:37:50 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:37:55 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:37:55 App: TestCasReplace got un-deployed
2021/12/08 11:37:55 No eventing-con process running
2021/12/08 11:37:55 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/08 11:37:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:38:07 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/08 11:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:38:14 Testing on the destination bucket for upsert with cas
2021/12/08 11:38:14 Starting bucket ops to source bucket
2021/12/08 11:38:14 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:38:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/08 11:38:20 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:38:25 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:38:30 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:38:35 Waiting for app: TestCasReplace to get deployed
2021/12/08 11:38:35 App: TestCasReplace got deployed
2021/12/08 11:38:35 Starting bucket ops to source bucket
2021/12/08 11:38:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:38:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:38:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:38:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:38:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:38:50 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:38:55 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:39:00 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:39:05 Waiting for app: TestCasReplace to get un-deployed
2021/12/08 11:39:05 App: TestCasReplace got un-deployed
2021/12/08 11:39:05 No eventing-con process running
2021/12/08 11:39:06 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/08 11:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:39:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:39:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:39:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.16s)
=== RUN   TestCasDelete
2021/12/08 11:39:24 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:39:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 11:39:30 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:39:35 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:39:40 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:39:45 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:39:45 App: TestCasDelete got deployed
2021/12/08 11:39:45 Starting bucket ops to source bucket
2021/12/08 11:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:39:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:39:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:39:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:39:51 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:39:56 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:40:01 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:40:06 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:40:06 App: TestCasDelete got un-deployed
2021/12/08 11:40:06 No eventing-con process running
2021/12/08 11:40:06 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/08 11:40:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:40:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:40:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:40:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:40:20 Testing on the destination bucket for delete with cas
2021/12/08 11:40:20 Starting bucket ops to source bucket
2021/12/08 11:40:20 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:40:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 11:40:26 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:40:31 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:40:36 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:40:41 Waiting for app: TestCasDelete to get deployed
2021/12/08 11:40:41 App: TestCasDelete got deployed
2021/12/08 11:40:41 Starting bucket ops to source bucket
2021/12/08 11:40:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:40:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:40:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:40:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:40:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 11:40:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:40:56 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:41:01 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:41:06 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:41:11 Waiting for app: TestCasDelete to get un-deployed
2021/12/08 11:41:11 App: TestCasDelete got un-deployed
2021/12/08 11:41:11 No eventing-con process running
2021/12/08 11:41:11 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/08 11:41:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:41:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:41:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:41:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:41:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.84s)
=== RUN   TestCountersIncrement
2021/12/08 11:41:35 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 11:41:41 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 11:41:41 Removing node(s): [] from the cluster
2021/12/08 11:41:46 Rebalance progress: 100
2021/12/08 11:41:46 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:41:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/08 11:41:53 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:41:58 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:42:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:42:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:42:08 App: TestCountersIncrement got deployed
2021/12/08 11:42:08 Starting bucket ops to source bucket
2021/12/08 11:42:14 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 11:42:19 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 11:42:24 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/08 11:42:24 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 11:42:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:42:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:42:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:42:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:42:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:42:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:42:44 App: TestCountersIncrement got un-deployed
2021/12/08 11:42:44 No eventing-con process running
2021/12/08 11:42:44 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/08 11:42:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:42:56 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/08 11:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:42:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:43:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/08 11:43:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:43:08 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:43:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/08 11:43:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:43:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:43:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:43:29 Waiting for app: TestCountersIncrement to get deployed
2021/12/08 11:43:29 App: TestCountersIncrement got deployed
2021/12/08 11:43:29 Starting bucket ops to source bucket
2021/12/08 11:43:35 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 11:43:40 Waiting for dst bucket item count to get to: 2048 curr count: 293
2021/12/08 11:43:45 Waiting for dst bucket item count to get to: 2048 curr count: 1317
2021/12/08 11:43:45 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 11:43:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:43:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:43:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:43:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:44:00 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:44:05 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/08 11:44:05 App: TestCountersIncrement got un-deployed
2021/12/08 11:44:05 No eventing-con process running
2021/12/08 11:44:05 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/08 11:44:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:44:17 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 11:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:44:29 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 11:44:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:44:29 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/08 11:44:34 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.90s)
=== RUN   TestCountersDecrement
2021/12/08 11:44:34 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 11:44:34 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 11:44:35 Removing node(s): [] from the cluster
2021/12/08 11:44:40 Rebalance progress: 100
2021/12/08 11:44:40 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:44:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/08 11:44:46 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:44:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:44:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:45:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:45:01 App: TestCountersDecrement got deployed
2021/12/08 11:45:01 Starting bucket ops to source bucket
2021/12/08 11:45:07 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 11:45:12 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/08 11:45:12 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/08 11:45:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:45:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:45:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:45:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:45:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:45:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:45:32 App: TestCountersDecrement got un-deployed
2021/12/08 11:45:32 No eventing-con process running
2021/12/08 11:45:32 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/08 11:45:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:45:44 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/08 11:45:49 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/08 11:45:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:45:51 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:46:01 Waiting for dst bucket item count to get to: 0 curr count: 936
2021/12/08 11:46:06 Waiting for dst bucket item count to get to: 0 curr count: 936
2021/12/08 11:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:46:06 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/08 11:46:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:46:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:46:22 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:46:27 Waiting for app: TestCountersDecrement to get deployed
2021/12/08 11:46:27 App: TestCountersDecrement got deployed
2021/12/08 11:46:27 Starting bucket ops to source bucket
2021/12/08 11:46:33 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/08 11:46:38 Waiting for dst bucket item count to get to: 2048 curr count: 1043
2021/12/08 11:46:43 Waiting for dst bucket item count to get to: 2048 curr count: 1043
2021/12/08 11:46:43 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/08 11:46:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:46:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:46:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:46:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:46:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:47:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/08 11:47:03 App: TestCountersDecrement got un-deployed
2021/12/08 11:47:03 No eventing-con process running
2021/12/08 11:47:03 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/08 11:47:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:47:15 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/08 11:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:47:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:47:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 11:47:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:47:27 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/08 11:47:32 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.23s)
=== RUN   TestCacheGet
2021/12/08 11:47:32 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:47:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:47:38 Waiting for app: TestCacheGet to get deployed
2021/12/08 11:47:43 Waiting for app: TestCacheGet to get deployed
2021/12/08 11:47:48 Waiting for app: TestCacheGet to get deployed
2021/12/08 11:47:53 Waiting for app: TestCacheGet to get deployed
2021/12/08 11:47:53 App: TestCacheGet got deployed
2021/12/08 11:47:53 Starting bucket ops to source bucket
2021/12/08 11:47:58 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 11:48:03 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 11:48:08 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/08 11:48:13 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/08 11:48:18 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/08 11:48:23 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/08 11:48:23 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/08 11:48:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:48:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:48:29 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 11:48:34 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 11:48:39 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 11:48:44 Waiting for app: TestCacheGet to get un-deployed
2021/12/08 11:48:44 App: TestCacheGet got un-deployed
2021/12/08 11:48:44 No eventing-con process running
2021/12/08 11:48:44 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/08 11:48:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:48:55 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/08 11:48:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:48:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:49:07 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/08 11:49:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.27s)
=== RUN   TestCacheRYOW
2021/12/08 11:49:07 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:49:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/08 11:49:14 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 11:49:19 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 11:49:24 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 11:49:29 Waiting for app: TestCacheRYOW to get deployed
2021/12/08 11:49:29 App: TestCacheRYOW got deployed
2021/12/08 11:49:29 Starting bucket ops to source bucket
2021/12/08 11:49:34 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/08 11:49:39 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/08 11:49:39 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/08 11:49:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:49:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:49:44 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 11:49:49 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 11:49:54 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 11:49:59 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/08 11:49:59 App: TestCacheRYOW got un-deployed
2021/12/08 11:49:59 No eventing-con process running
2021/12/08 11:49:59 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/08 11:50:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:50:11 Waiting for dst bucket item count to get to: 0 curr count: 45
2021/12/08 11:50:16 Waiting for dst bucket item count to get to: 0 curr count: 45
2021/12/08 11:50:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:50:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:50:28 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/08 11:50:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.24s)
=== RUN   TestCacheExpiry
2021/12/08 11:50:28 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:50:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/08 11:50:34 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 11:50:39 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 11:50:44 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 11:50:49 Waiting for app: TestCacheExpiry to get deployed
2021/12/08 11:50:49 App: TestCacheExpiry got deployed
2021/12/08 11:50:49 Starting bucket ops to source bucket
2021/12/08 11:50:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:50:59 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/08 11:51:04 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/08 11:51:09 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/08 11:51:14 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/08 11:51:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:51:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:51:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:51:19 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 11:51:24 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 11:51:29 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 11:51:34 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/08 11:51:34 App: TestCacheExpiry got un-deployed
2021/12/08 11:51:34 No eventing-con process running
2021/12/08 11:51:34 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/08 11:51:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:51:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:51:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:51:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:51:58 Waiting for dst bucket item count to get to: 0 curr count: 63
2021/12/08 11:52:03 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/08 11:52:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (95.33s)
=== RUN   TestCacheExpiryCustom
2021/12/08 11:52:03 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:52:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/08 11:52:09 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 11:52:14 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 11:52:19 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 11:52:24 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/08 11:52:24 App: TestCacheExpiryCustom got deployed
2021/12/08 11:52:24 Starting bucket ops to source bucket
2021/12/08 11:52:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:52:34 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/08 11:52:39 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/08 11:52:44 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/08 11:52:49 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/08 11:52:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:52:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:52:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:52:55 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 11:53:00 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 11:53:05 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 11:53:10 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/08 11:53:10 App: TestCacheExpiryCustom got un-deployed
2021/12/08 11:53:10 No eventing-con process running
2021/12/08 11:53:10 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/08 11:53:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:53:21 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:53:26 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:53:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:53:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:53:38 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/08 11:53:43 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/08 11:53:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.29s)
=== RUN   TestCacheOverflowCustom
2021/12/08 11:53:43 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:53:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/08 11:53:49 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 11:53:54 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 11:53:59 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 11:54:04 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/08 11:54:04 App: TestCacheOverflowCustom got deployed
2021/12/08 11:54:04 Starting bucket ops to source bucket
2021/12/08 11:54:09 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/08 11:54:15 Waiting for dst bucket item count to get to: 35 curr count: 7
2021/12/08 11:54:20 Waiting for dst bucket item count to get to: 35 curr count: 16
2021/12/08 11:54:25 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/08 11:54:30 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/08 11:54:35 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/08 11:54:35 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/08 11:54:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:54:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:54:40 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 11:54:45 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 11:54:50 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 11:54:55 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/08 11:54:55 App: TestCacheOverflowCustom got un-deployed
2021/12/08 11:54:55 No eventing-con process running
2021/12/08 11:54:55 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/08 11:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:55:07 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/08 11:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:55:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:55:18 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/08 11:55:23 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/08 11:55:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.24s)
=== RUN   TestStrictMode
2021/12/08 11:55:23 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:55:24 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/08 11:55:24 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:55:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/08 11:55:31 Waiting for app: TestDataTypes to get deployed
2021/12/08 11:55:36 Waiting for app: TestDataTypes to get deployed
2021/12/08 11:55:41 Waiting for app: TestDataTypes to get deployed
2021/12/08 11:55:46 Waiting for app: TestDataTypes to get deployed
2021/12/08 11:55:46 App: TestDataTypes got deployed
2021/12/08 11:55:46 Starting bucket ops to source bucket
2021/12/08 11:55:51 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/08 11:55:56 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/08 11:55:56 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/08 11:55:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:55:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:56:01 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 11:56:06 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 11:56:11 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 11:56:16 Waiting for app: TestDataTypes to get un-deployed
2021/12/08 11:56:16 App: TestDataTypes got un-deployed
2021/12/08 11:56:16 No eventing-con process running
2021/12/08 11:56:16 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/08 11:56:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:56:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:56:35 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/08 11:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.63s)
=== RUN   TestEnoentGet
2021/12/08 11:56:35 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:56:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:56:41 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:56:46 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:56:51 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:56:56 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:56:56 App: TestEnoentGet got deployed
2021/12/08 11:56:56 Starting bucket ops to source bucket
2021/12/08 11:57:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:57:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:57:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:57:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:57:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:57:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:57:17 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:57:22 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:57:22 App: TestEnoentGet got un-deployed
2021/12/08 11:57:22 No eventing-con process running
2021/12/08 11:57:22 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/08 11:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:57:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:57:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:57:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:57:50 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/08 11:57:55 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/08 11:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:57:55 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/08 11:58:01 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:58:06 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:58:11 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:58:16 Waiting for app: TestEnoentGet to get deployed
2021/12/08 11:58:16 App: TestEnoentGet got deployed
2021/12/08 11:58:16 Starting bucket ops to source bucket
2021/12/08 11:58:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:58:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:58:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:58:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:58:27 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:58:32 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:58:37 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:58:42 Waiting for app: TestEnoentGet to get un-deployed
2021/12/08 11:58:42 App: TestEnoentGet got un-deployed
2021/12/08 11:58:42 No eventing-con process running
2021/12/08 11:58:42 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/08 11:58:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:58:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 11:58:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 11:58:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 11:59:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 11:59:10 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/08 11:59:15 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/08 11:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.41s)
=== RUN   TestEnoentDelete
2021/12/08 11:59:15 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 11:59:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 11:59:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 11:59:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 11:59:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 11:59:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 11:59:37 App: TestEnoentDelete got deployed
2021/12/08 11:59:37 Starting bucket ops to source bucket
2021/12/08 11:59:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 11:59:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 11:59:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 11:59:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 11:59:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 11:59:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 11:59:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:00:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:00:02 App: TestEnoentDelete got un-deployed
2021/12/08 12:00:02 No eventing-con process running
2021/12/08 12:00:02 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/08 12:00:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:00:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 12:00:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 12:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:00:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:00:31 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/08 12:00:36 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/08 12:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:00:36 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:00:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 12:00:42 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 12:00:47 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 12:00:52 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 12:00:57 Waiting for app: TestEnoentDelete to get deployed
2021/12/08 12:00:57 App: TestEnoentDelete got deployed
2021/12/08 12:00:57 Starting bucket ops to source bucket
2021/12/08 12:01:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 12:01:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 12:01:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:01:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:01:07 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:01:12 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:01:17 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:01:22 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/08 12:01:22 App: TestEnoentDelete got un-deployed
2021/12/08 12:01:22 No eventing-con process running
2021/12/08 12:01:22 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/08 12:01:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:01:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 12:01:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 12:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:01:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:01:51 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/08 12:01:56 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/08 12:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.36s)
=== RUN   TestError
2021/12/08 12:02:06 Waiting for app: TestError to get un-deployed
2021/12/08 12:02:06 App: TestError got un-deployed
2021/12/08 12:02:06 No eventing-con process running
2021/12/08 12:02:06 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/08 12:02:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:02:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:02:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:02:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:02:20 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:02:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/08 12:02:26 Waiting for app: TestError to get deployed
2021/12/08 12:02:31 Waiting for app: TestError to get deployed
2021/12/08 12:02:36 Waiting for app: TestError to get deployed
2021/12/08 12:02:41 Waiting for app: TestError to get deployed
2021/12/08 12:02:41 App: TestError got deployed
2021/12/08 12:02:41 Starting bucket ops to source bucket
2021/12/08 12:02:46 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/08 12:02:51 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/08 12:02:51 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/08 12:02:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:02:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:02:56 Waiting for app: TestError to get un-deployed
2021/12/08 12:03:01 Waiting for app: TestError to get un-deployed
2021/12/08 12:03:06 Waiting for app: TestError to get un-deployed
2021/12/08 12:03:11 Waiting for app: TestError to get un-deployed
2021/12/08 12:03:11 App: TestError got un-deployed
2021/12/08 12:03:11 No eventing-con process running
2021/12/08 12:03:11 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/08 12:03:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:03:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 12:03:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/08 12:03:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:03:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:03:40 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/08 12:03:45 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/08 12:03:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.89s)
=== RUN   TestCRLF
2021/12/08 12:03:55 Waiting for app: TestCRLF to get un-deployed
2021/12/08 12:03:55 App: TestCRLF got un-deployed
2021/12/08 12:03:55 No eventing-con process running
2021/12/08 12:03:55 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/08 12:03:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:04:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:04:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:04:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:04:08 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:04:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/08 12:04:14 Waiting for app: TestCRLF to get deployed
2021/12/08 12:04:19 Waiting for app: TestCRLF to get deployed
2021/12/08 12:04:24 Waiting for app: TestCRLF to get deployed
2021/12/08 12:04:29 Waiting for app: TestCRLF to get deployed
2021/12/08 12:04:30 App: TestCRLF got deployed
2021/12/08 12:04:30 Starting bucket ops to source bucket
2021/12/08 12:04:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 12:04:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 12:04:45 Waiting for dst bucket item count to get to: 100 curr count: 52
2021/12/08 12:04:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 12:04:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:04:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:04:50 Waiting for app: TestCRLF to get un-deployed
2021/12/08 12:04:55 Waiting for app: TestCRLF to get un-deployed
2021/12/08 12:05:00 Waiting for app: TestCRLF to get un-deployed
2021/12/08 12:05:05 Waiting for app: TestCRLF to get un-deployed
2021/12/08 12:05:05 App: TestCRLF got un-deployed
2021/12/08 12:05:05 No eventing-con process running
2021/12/08 12:05:05 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/08 12:05:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:05:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 12:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:05:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:05:29 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/08 12:05:34 Waiting for dst bucket item count to get to: 0 curr count: 38
2021/12/08 12:05:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.82s)
=== RUN   TestImportExport
2021/12/08 12:05:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:05:44 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 12:05:44 No eventing-con process running
2021/12/08 12:05:44 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/08 12:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:05:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:05:52 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:06:02 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:06:02 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 12:06:02 No eventing-con process running
2021/12/08 12:06:02 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/08 12:06:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:06:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:06:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:06:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:06:16 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:06:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 12:06:17 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:06:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 12:06:23 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 12:06:28 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 12:06:33 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 12:06:38 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/08 12:06:38 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/08 12:06:43 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 12:06:48 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 12:06:53 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/08 12:06:53 App: TestImportExport_bucket_op_on_update got deployed
2021/12/08 12:06:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:06:58 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:07:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:07:08 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:07:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:07:13 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 12:07:13 No eventing-con process running
2021/12/08 12:07:13 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/08 12:07:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:07:19 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:07:24 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:07:29 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:07:34 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:07:34 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 12:07:34 No eventing-con process running
2021/12/08 12:07:34 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/08 12:07:51 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/08 12:07:51 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/08 12:07:51 No eventing-con process running
2021/12/08 12:07:51 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/08 12:07:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:07:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:08:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:08:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:08:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/08 12:08:10 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/08 12:08:10 No eventing-con process running
2021/12/08 12:08:10 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/08 12:08:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:08:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:08:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.67s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/08 12:08:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:08:33 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:08:33 No eventing-con process running
2021/12/08 12:08:33 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/08 12:08:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:08:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:08:42 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:08:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:08:47 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:08:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:08:48 Starting bucket ops to source bucket
2021/12/08 12:08:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:09:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:09:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:09:11 Waiting for dst bucket item count to get to: 5000 curr count: 134
2021/12/08 12:09:16 Waiting for dst bucket item count to get to: 5000 curr count: 134
2021/12/08 12:09:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:09:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 12:09:21 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 12:09:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:09:21 Undeploying app: bucket_op_on_update
2021/12/08 12:09:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:09:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:09:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:09:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:09:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:09:41 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:09:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:09:41 No eventing-con process running
2021/12/08 12:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:10:25 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:10:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:10:26 Starting bucket ops to source bucket
2021/12/08 12:10:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:10:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:10:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:10:49 Waiting for dst bucket item count to get to: 5000 curr count: 1783
2021/12/08 12:10:54 Waiting for dst bucket item count to get to: 5000 curr count: 1783
2021/12/08 12:10:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:10:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 12:10:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 12:10:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:10:59 Undeploying app: bucket_op_on_update
2021/12/08 12:11:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:11:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:11:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:11:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:11:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:11:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:11:20 No eventing-con process running
2021/12/08 12:11:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:11:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:12:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:12:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:12:05 Starting bucket ops to source bucket
2021/12/08 12:12:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:12:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:12:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:12:28 Waiting for dst bucket item count to get to: 5000 curr count: 676
2021/12/08 12:12:33 Waiting for dst bucket item count to get to: 5000 curr count: 3176
2021/12/08 12:12:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:12:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 12:12:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 12:12:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:12:38 Undeploying app: bucket_op_on_update
2021/12/08 12:12:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:12:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:12:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:12:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:12:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:12:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:12:58 No eventing-con process running
2021/12/08 12:13:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:13:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:13:42 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:13:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:13:43 Starting bucket ops to source bucket
2021/12/08 12:13:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:13:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:14:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:14:06 Waiting for dst bucket item count to get to: 5000 curr count: 2108
2021/12/08 12:14:11 Waiting for dst bucket item count to get to: 5000 curr count: 4608
2021/12/08 12:14:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:14:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 12:14:16 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 12:14:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:14:16 Undeploying app: bucket_op_on_update
2021/12/08 12:14:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:14:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:14:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:14:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:14:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:14:36 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:14:36 No eventing-con process running
2021/12/08 12:14:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:15:20 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:15:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:15:22 Starting bucket ops to source bucket
2021/12/08 12:15:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:15:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:15:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:15:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:15:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:15:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/08 12:15:50 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/08 12:15:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:15:50 Undeploying app: bucket_op_on_update
2021/12/08 12:15:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:15:55 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:16:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:16:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:16:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/08 12:16:10 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/08 12:16:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:16:10 No eventing-con process running
2021/12/08 12:16:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:16:54 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.64s)
=== RUN   TestOnUpdateN1QLOp
2021/12/08 12:17:04 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 12:17:04 App: TestOnUpdateN1QLOp got un-deployed
2021/12/08 12:17:04 No eventing-con process running
2021/12/08 12:17:04 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/08 12:17:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:17:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:17:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:17:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:17:17 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:17:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/08 12:17:19 Starting bucket ops to source bucket
2021/12/08 12:17:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:17:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:17:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:17:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:17:47 Waiting for dst bucket item count to get to: 5000 curr count: 2546
2021/12/08 12:17:47 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:17:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:17:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:17:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 12:17:57 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 12:18:02 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 12:18:07 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/08 12:18:07 App: TestOnUpdateN1QLOp got un-deployed
2021/12/08 12:18:07 No eventing-con process running
2021/12/08 12:18:07 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/08 12:18:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:18:18 Waiting for dst bucket item count to get to: 0 curr count: 4204
2021/12/08 12:18:23 Waiting for dst bucket item count to get to: 0 curr count: 1704
2021/12/08 12:18:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:18:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:18:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:18:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.38s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/08 12:18:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 12:18:45 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/08 12:18:45 No eventing-con process running
2021/12/08 12:18:45 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/08 12:18:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:18:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:18:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:18:59 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:19:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:19:00 Starting bucket ops to source bucket
2021/12/08 12:19:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:19:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:19:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:19:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:19:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/08 12:19:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:19:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:19:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:19:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 12:19:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 12:19:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 12:19:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/08 12:19:48 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/08 12:19:48 No eventing-con process running
2021/12/08 12:19:48 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/08 12:19:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:20:00 Waiting for dst bucket item count to get to: 0 curr count: 2916
2021/12/08 12:20:05 Waiting for dst bucket item count to get to: 0 curr count: 2916
2021/12/08 12:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:20:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:20:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:20:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.52s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/08 12:20:27 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 12:20:27 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 12:20:27 No eventing-con process running
2021/12/08 12:20:27 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/08 12:20:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:20:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:20:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:20:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:20:40 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:20:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:20:41 Starting bucket ops to source bucket
2021/12/08 12:20:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:20:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:20:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:21:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:21:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:21:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:21:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:21:10 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 12:21:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 12:21:20 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 12:21:25 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 12:21:25 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 12:21:25 No eventing-con process running
2021/12/08 12:21:25 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/08 12:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:21:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:21:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:21:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:21:48 Waiting for dst bucket item count to get to: 0 curr count: 4093
2021/12/08 12:21:53 Waiting for dst bucket item count to get to: 0 curr count: 1593
2021/12/08 12:21:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.58s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/08 12:22:03 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 12:22:03 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/08 12:22:03 No eventing-con process running
2021/12/08 12:22:03 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/08 12:22:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:22:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:22:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:22:17 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:22:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/08 12:22:18 Starting bucket ops to source bucket
2021/12/08 12:22:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:22:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:22:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:22:41 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/08 12:22:46 Waiting for dst bucket item count to get to: 5000 curr count: 67
2021/12/08 12:22:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:22:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:22:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:22:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 12:22:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 12:23:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 12:23:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/08 12:23:06 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/08 12:23:06 No eventing-con process running
2021/12/08 12:23:06 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/08 12:23:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:23:18 Waiting for dst bucket item count to get to: 0 curr count: 3422
2021/12/08 12:23:23 Waiting for dst bucket item count to get to: 0 curr count: 922
2021/12/08 12:23:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:23:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:23:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.56s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/08 12:23:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 12:23:45 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/08 12:23:45 No eventing-con process running
2021/12/08 12:23:45 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/08 12:23:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:23:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:23:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:23:59 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:24:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/08 12:24:00 Starting bucket ops to source bucket
2021/12/08 12:24:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:24:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:24:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:24:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:24:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/08 12:24:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:24:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:24:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:24:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 12:24:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 12:24:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 12:24:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/08 12:24:48 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/08 12:24:48 No eventing-con process running
2021/12/08 12:24:48 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/08 12:24:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:25:00 Waiting for dst bucket item count to get to: 0 curr count: 3448
2021/12/08 12:25:05 Waiting for dst bucket item count to get to: 0 curr count: 3448
2021/12/08 12:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:25:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.21s)
=== RUN   TestOnDeleteBucketOp
2021/12/08 12:25:27 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 12:25:27 App: TestOnDeleteBucketOp got un-deployed
2021/12/08 12:25:27 No eventing-con process running
2021/12/08 12:25:27 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/08 12:25:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:25:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:25:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:25:41 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:25:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 12:25:42 Starting bucket ops to source bucket
2021/12/08 12:25:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:25:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:26:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:26:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:26:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:26:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:26:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 12:26:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 12:26:18 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 12:26:23 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/08 12:26:23 App: TestOnDeleteBucketOp got un-deployed
2021/12/08 12:26:23 No eventing-con process running
2021/12/08 12:26:23 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/08 12:26:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:26:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:26:32 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:26:42 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:26:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:26:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.80s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/08 12:26:57 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 12:26:57 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/08 12:26:57 No eventing-con process running
2021/12/08 12:26:57 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/08 12:26:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:27:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:27:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:27:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:27:11 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:27:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/08 12:27:12 Starting bucket ops to source bucket
2021/12/08 12:27:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:27:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:27:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:27:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:27:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:27:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:27:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 12:27:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 12:27:48 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 12:27:53 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/08 12:27:53 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/08 12:27:53 No eventing-con process running
2021/12/08 12:27:53 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/08 12:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:28:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:28:12 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:28:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:28:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.03s)
=== RUN   TestTimerBucketOp
2021/12/08 12:28:27 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 12:28:27 App: TestTimerBucketOp got un-deployed
2021/12/08 12:28:27 No eventing-con process running
2021/12/08 12:28:27 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/08 12:28:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:28:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:28:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:28:41 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:28:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 12:28:42 Starting bucket ops to source bucket
2021/12/08 12:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:29:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:29:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:29:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:29:20 Waiting for dst bucket item count to get to: 5000 curr count: 1272
2021/12/08 12:29:25 Waiting for dst bucket item count to get to: 5000 curr count: 1272
2021/12/08 12:29:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:29:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:29:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:29:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 12:29:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 12:29:40 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 12:29:45 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/08 12:29:45 App: TestTimerBucketOp got un-deployed
2021/12/08 12:29:45 No eventing-con process running
2021/12/08 12:29:45 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/08 12:29:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:29:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:29:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:29:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:30:08 Waiting for dst bucket item count to get to: 0 curr count: 4350
2021/12/08 12:30:13 Waiting for dst bucket item count to get to: 0 curr count: 1850
2021/12/08 12:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.50s)
=== RUN   TestTimerInPastBucketOp
2021/12/08 12:30:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 12:30:23 App: TestTimerInPastBucketOp got un-deployed
2021/12/08 12:30:23 No eventing-con process running
2021/12/08 12:30:23 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/08 12:30:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:30:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:30:32 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:30:37 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:30:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 12:30:38 Starting bucket ops to source bucket
2021/12/08 12:30:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:30:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:30:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:31:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:31:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:31:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:31:16 Waiting for dst bucket item count to get to: 5000 curr count: 2443
2021/12/08 12:31:21 Waiting for dst bucket item count to get to: 5000 curr count: 4943
2021/12/08 12:31:21 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:31:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:31:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:31:26 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 12:31:31 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 12:31:36 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 12:31:41 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/08 12:31:41 App: TestTimerInPastBucketOp got un-deployed
2021/12/08 12:31:41 No eventing-con process running
2021/12/08 12:31:41 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/08 12:31:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:31:53 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 12:31:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 12:31:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:32:10 Waiting for dst bucket item count to get to: 0 curr count: 3263
2021/12/08 12:32:15 Waiting for dst bucket item count to get to: 0 curr count: 3263
2021/12/08 12:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.60s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/08 12:32:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:32:25 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:32:25 No eventing-con process running
2021/12/08 12:32:25 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/08 12:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:32:39 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:32:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 12:32:40 Starting bucket ops to source bucket
2021/12/08 12:32:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:32:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:32:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:33:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:33:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:33:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:33:18 Waiting for dst bucket item count to get to: 5000 curr count: 932
2021/12/08 12:33:23 Waiting for dst bucket item count to get to: 5000 curr count: 3432
2021/12/08 12:33:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:33:28 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 12:33:28 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 12:33:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:33:28 Undeploying app: bucket_op_with_timer
2021/12/08 12:33:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:33:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:33:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:33:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:33:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:33:48 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:33:48 No eventing-con process running
2021/12/08 12:33:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:34:32 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:34:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 12:34:33 Starting bucket ops to source bucket
2021/12/08 12:34:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:34:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:34:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:34:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:35:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:35:06 Waiting for dst bucket item count to get to: 5000 curr count: 801
2021/12/08 12:35:11 Waiting for dst bucket item count to get to: 5000 curr count: 1932
2021/12/08 12:35:16 Waiting for dst bucket item count to get to: 5000 curr count: 3631
2021/12/08 12:35:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:35:21 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 12:35:21 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 12:35:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:35:21 Undeploying app: bucket_op_with_timer
2021/12/08 12:35:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:35:26 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:35:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:35:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:35:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:35:41 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:35:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:35:41 No eventing-con process running
2021/12/08 12:35:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:35:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:36:25 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:36:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 12:36:27 Starting bucket ops to source bucket
2021/12/08 12:36:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:36:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:36:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:36:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:36:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:36:59 Waiting for dst bucket item count to get to: 5000 curr count: 517
2021/12/08 12:37:05 Waiting for dst bucket item count to get to: 5000 curr count: 517
2021/12/08 12:37:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:37:10 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 12:37:10 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 12:37:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:37:10 Undeploying app: bucket_op_with_timer
2021/12/08 12:37:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:37:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:37:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:37:25 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:37:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:37:30 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:37:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:37:30 No eventing-con process running
2021/12/08 12:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:37:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:38:14 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:38:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 12:38:15 Starting bucket ops to source bucket
2021/12/08 12:38:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:38:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:38:58 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 12:38:58 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 12:38:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:38:58 Undeploying app: bucket_op_with_timer
2021/12/08 12:38:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:39:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:39:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:39:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:39:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:39:18 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:39:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:39:18 No eventing-con process running
2021/12/08 12:39:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:39:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:40:02 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:40:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 12:40:03 Starting bucket ops to source bucket
2021/12/08 12:40:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:40:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:40:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:40:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:40:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:40:36 Waiting for dst bucket item count to get to: 5000 curr count: 279
2021/12/08 12:40:41 Waiting for dst bucket item count to get to: 5000 curr count: 946
2021/12/08 12:40:46 Waiting for dst bucket item count to get to: 5000 curr count: 3167
2021/12/08 12:40:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:40:51 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/08 12:40:51 App: TestDeployUndeployLoopTimer got deployed
2021/12/08 12:40:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:40:51 Undeploying app: bucket_op_with_timer
2021/12/08 12:40:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:40:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:41:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:41:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:41:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/08 12:41:11 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/08 12:41:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:41:11 No eventing-con process running
2021/12/08 12:41:13 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:41:55 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.20s)
=== RUN   TestMultipleHandlers
2021/12/08 12:42:05 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 12:42:05 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/08 12:42:05 No eventing-con process running
2021/12/08 12:42:05 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/08 12:42:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:42:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:42:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:42:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:42:24 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 12:42:24 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/08 12:42:24 No eventing-con process running
2021/12/08 12:42:24 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/08 12:42:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:42:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:42:32 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:42:38 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:42:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 12:42:39 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:42:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/08 12:42:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 12:42:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 12:42:55 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 12:43:00 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/08 12:43:00 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/08 12:43:05 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 12:43:10 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 12:43:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/08 12:43:15 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/08 12:43:15 Starting bucket ops to source bucket
2021/12/08 12:43:25 Waiting for dst bucket item count to get to: 5000 curr count: 1164
2021/12/08 12:43:30 Waiting for dst bucket item count to get to: 5000 curr count: 3398
2021/12/08 12:43:35 Waiting for dst bucket item count to get to: 5000 curr count: 4734
2021/12/08 12:43:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:43:35 Starting bucket ops to source bucket
2021/12/08 12:43:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:43:55 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/08 12:43:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 12:43:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:43:55 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:43:55 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:43:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:44:00 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 12:44:05 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 12:44:10 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 12:44:15 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/08 12:44:15 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/08 12:44:15 No eventing-con process running
2021/12/08 12:44:15 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/08 12:44:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:44:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:44:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:44:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 12:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:44:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:44:39 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 12:44:44 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 12:44:49 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 12:44:54 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/08 12:44:54 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/08 12:44:55 No eventing-con process running
2021/12/08 12:44:55 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/08 12:44:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:45:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:45:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:45:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.87s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/08 12:45:18 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 12:45:18 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/08 12:45:18 No eventing-con process running
2021/12/08 12:45:18 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/08 12:45:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:45:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:45:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:45:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:45:32 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:45:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:45:33 Starting bucket ops to source bucket
2021/12/08 12:45:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:45:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:45:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:45:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:45:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:45:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:46:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 12:46:01 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 12:46:01 Pausing app: bucket_op_on_update
2021/12/08 12:46:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:46:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 12:46:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 12:46:06 Resuming app: bucket_op_on_update
2021/12/08 12:46:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:46:07 Starting bucket ops to source bucket
2021/12/08 12:46:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:46:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:46:25 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/08 12:46:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 12:46:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:46:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 12:46:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 12:46:30 Pausing app: bucket_op_on_update
2021/12/08 12:46:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:46:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 12:46:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 12:46:35 Resuming app: bucket_op_on_update
2021/12/08 12:46:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:46:36 Starting bucket ops to source bucket
2021/12/08 12:46:44 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:46:49 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:46:54 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/08 12:46:54 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/08 12:46:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:46:59 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 12:46:59 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 12:46:59 Pausing app: bucket_op_on_update
2021/12/08 12:47:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:47:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 12:47:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 12:47:05 Resuming app: bucket_op_on_update
2021/12/08 12:47:06 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:47:06 Starting bucket ops to source bucket
2021/12/08 12:47:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:47:19 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:47:24 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:47:24 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 12:47:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:47:29 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 12:47:29 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 12:47:29 Pausing app: bucket_op_on_update
2021/12/08 12:47:29 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:47:34 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 12:47:34 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 12:47:34 Resuming app: bucket_op_on_update
2021/12/08 12:47:35 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:47:35 Starting bucket ops to source bucket
2021/12/08 12:47:43 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:47:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:47:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:47:58 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/08 12:47:58 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 12:47:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:48:03 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/08 12:48:03 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/08 12:48:03 Pausing app: bucket_op_on_update
2021/12/08 12:48:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:48:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/08 12:48:08 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/08 12:48:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:48:13 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 12:48:18 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 12:48:23 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 12:48:28 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/08 12:48:28 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/08 12:48:28 No eventing-con process running
2021/12/08 12:48:28 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/08 12:48:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:48:40 Waiting for dst bucket item count to get to: 0 curr count: 15312
2021/12/08 12:48:45 Waiting for dst bucket item count to get to: 0 curr count: 15312
2021/12/08 12:48:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:48:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:48:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 12:48:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (228.79s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/08 12:49:07 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 12:49:07 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/08 12:49:07 No eventing-con process running
2021/12/08 12:49:07 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/08 12:49:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:49:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:49:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:49:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:49:21 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:49:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 12:49:22 Starting bucket ops to source bucket
2021/12/08 12:49:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:49:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:49:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:49:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:49:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:49:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:50:00 Waiting for dst bucket item count to get to: 5000 curr count: 1787
2021/12/08 12:50:05 Waiting for dst bucket item count to get to: 5000 curr count: 1787
2021/12/08 12:50:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:50:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:50:10 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 12:50:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 12:50:10 Pausing app: bucket_op_with_timer
2021/12/08 12:50:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:50:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 12:50:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 12:50:15 Resuming app: bucket_op_with_timer
2021/12/08 12:50:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:50:16 Starting bucket ops to source bucket
2021/12/08 12:50:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:50:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:50:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:50:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:50:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:50:49 Waiting for dst bucket item count to get to: 10000 curr count: 7267
2021/12/08 12:50:54 Waiting for dst bucket item count to get to: 10000 curr count: 7267
2021/12/08 12:50:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 12:50:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:50:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 12:50:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 12:50:59 Pausing app: bucket_op_with_timer
2021/12/08 12:50:59 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:51:04 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 12:51:04 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 12:51:04 Resuming app: bucket_op_with_timer
2021/12/08 12:51:05 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:51:05 Starting bucket ops to source bucket
2021/12/08 12:51:13 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:51:18 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:51:23 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:51:28 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/08 12:51:33 Waiting for dst bucket item count to get to: 15000 curr count: 11869
2021/12/08 12:51:38 Waiting for dst bucket item count to get to: 15000 curr count: 14087
2021/12/08 12:51:43 Waiting for dst bucket item count to get to: 15000 curr count: 14718
2021/12/08 12:51:43 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/08 12:51:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:51:48 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 12:51:48 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 12:51:48 Pausing app: bucket_op_with_timer
2021/12/08 12:51:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:51:54 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 12:51:54 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 12:51:54 Resuming app: bucket_op_with_timer
2021/12/08 12:51:55 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:51:55 Starting bucket ops to source bucket
2021/12/08 12:52:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:52:08 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:52:13 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:52:18 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/08 12:52:23 Waiting for dst bucket item count to get to: 20000 curr count: 15493
2021/12/08 12:52:28 Waiting for dst bucket item count to get to: 20000 curr count: 17993
2021/12/08 12:52:28 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 12:52:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:52:33 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 12:52:33 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 12:52:33 Pausing app: bucket_op_with_timer
2021/12/08 12:52:33 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:52:38 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 12:52:38 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 12:52:38 Resuming app: bucket_op_with_timer
2021/12/08 12:52:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:52:39 Starting bucket ops to source bucket
2021/12/08 12:52:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:52:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:52:57 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:53:02 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:53:07 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:53:12 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/08 12:53:17 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/08 12:53:17 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 12:53:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:53:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/08 12:53:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/08 12:53:22 Pausing app: bucket_op_with_timer
2021/12/08 12:53:22 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:53:27 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/08 12:53:27 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/08 12:53:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:53:32 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 12:53:37 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 12:53:42 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 12:53:47 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/08 12:53:47 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/08 12:53:47 No eventing-con process running
2021/12/08 12:53:47 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/08 12:53:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:53:59 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 12:54:04 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 12:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:54:16 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 12:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.75s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/08 12:54:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 12:54:26 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/08 12:54:26 No eventing-con process running
2021/12/08 12:54:26 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/08 12:54:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:54:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:54:34 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:54:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:54:39 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:54:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/08 12:54:45 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 12:54:50 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 12:54:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 12:55:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/08 12:55:00 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/08 12:55:00 Starting bucket ops to source bucket
2021/12/08 12:55:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:55:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:55:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:55:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:55:30 Waiting for dst bucket item count to get to: 5000 curr count: 4876
2021/12/08 12:55:35 Waiting for dst bucket item count to get to: 5000 curr count: 4876
2021/12/08 12:55:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:55:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:55:35 Pausing app: bucket_op_with_timer
2021/12/08 12:55:35 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:55:40 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/08 12:55:40 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/08 12:55:40 Resuming app: bucket_op_with_timer
2021/12/08 12:55:41 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:55:41 Starting bucket ops to source bucket
2021/12/08 12:55:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:55:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:56:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:56:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:56:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:56:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 12:56:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 12:56:18 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 12:56:23 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 12:56:28 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 12:56:33 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/08 12:56:33 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/08 12:56:33 No eventing-con process running
2021/12/08 12:56:33 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/08 12:56:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:56:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 12:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:56:46 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:56:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 12:56:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.61s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/08 12:57:06 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 12:57:06 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/08 12:57:06 No eventing-con process running
2021/12/08 12:57:06 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/08 12:57:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:57:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:57:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 12:57:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 12:57:20 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 12:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 12:57:26 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 12:57:31 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 12:57:36 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 12:57:41 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/08 12:57:41 App: TestPauseResumeWithEventingReb got deployed
2021/12/08 12:57:41 Starting bucket ops to source bucket
2021/12/08 12:57:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:57:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:58:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:58:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 12:58:11 Waiting for dst bucket item count to get to: 5000 curr count: 4947
2021/12/08 12:58:16 Waiting for dst bucket item count to get to: 5000 curr count: 4947
2021/12/08 12:58:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 12:58:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 12:58:16 Pausing app: bucket_op_with_timer
2021/12/08 12:58:16 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:58:21 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/08 12:58:21 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/08 12:58:21 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 12:58:21 Removing node(s): [] from the cluster
2021/12/08 12:58:26 Rebalance progress: 29.16666666666666
2021/12/08 12:58:31 Rebalance progress: 29.16666666666666
2021/12/08 12:58:36 Rebalance progress: 100
2021/12/08 12:58:36 Resuming app: bucket_op_with_timer
2021/12/08 12:58:37 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 12:58:37 Starting bucket ops to source bucket
2021/12/08 12:58:49 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:58:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:58:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:59:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/08 12:59:09 Waiting for dst bucket item count to get to: 10000 curr count: 5504
2021/12/08 12:59:14 Waiting for dst bucket item count to get to: 10000 curr count: 5504
2021/12/08 12:59:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 12:59:14 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 12:59:19 Rebalance progress: 29.16666666666666
2021/12/08 12:59:24 Rebalance progress: 29.16666666666666
2021/12/08 12:59:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/08 12:59:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 12:59:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:00:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:00:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:00:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:00:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:01:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:01:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:01:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 13:01:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:71} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: c64ffcd42db24688dd2d82b8fefed45a
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
Producer node: 127.0.0.1:9302	Node UUID: 50ec84f50f20f053bef5ba2d7a27244e
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 3251124449281975ff2a3ddd4f098ca1
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]


2021/12/08 13:02:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:02:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:02:09 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 13:02:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 13:02:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 13:02:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/08 13:02:24 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/08 13:02:24 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:02:24 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/08 13:02:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:02:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:02:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:02:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:02:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6971
2021/12/08 13:02:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1971
2021/12/08 13:02:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (356.70s)
=== RUN   TestBucketDeleteAfterPause
2021/12/08 13:03:03 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/08 13:03:03 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/08 13:03:03 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:03:03 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/08 13:03:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:03:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:03:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:03:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:03:17 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:03:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/08 13:03:23 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 13:03:28 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 13:03:33 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 13:03:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/08 13:03:38 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/08 13:03:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:03:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 793
2021/12/08 13:03:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 793
2021/12/08 13:03:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 793
2021/12/08 13:03:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:03:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:03:57 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/08 13:03:58 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:04:03 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/08 13:04:03 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/08 13:04:03 handler_test.go:716: Deleting source bucket: default
2021/12/08 13:04:09 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/08 13:04:09 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/08 13:04:14 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/08 13:04:14 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/08 13:04:14 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:04:14 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/08 13:04:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:04:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 13:04:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:04:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:04:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:04:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (104.05s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/08 13:04:47 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 13:04:47 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/08 13:04:47 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:04:47 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/08 13:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:04:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:04:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:05:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:05:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:05:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/08 13:05:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 13:05:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 13:05:16 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 13:05:21 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/08 13:05:21 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/08 13:05:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:05:56 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/08 13:05:56 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:06:01 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/08 13:06:01 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/08 13:09:01 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/08 13:09:01 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/08 13:09:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/08 13:09:07 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 13:09:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:09:07 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/08 13:09:12 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 13:09:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:09:12 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3060 expected 5000
2021/12/08 13:09:17 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/08 13:09:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:09:18 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/08 13:09:18 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/08 13:09:18 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:09:28 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 13:09:33 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 13:09:38 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/08 13:09:38 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/08 13:09:38 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:09:38 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/08 13:09:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:09:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:09:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 13:09:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:09:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:10:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (323.71s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/08 13:10:11 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 13:10:11 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/08 13:10:11 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:10:11 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/08 13:10:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:10:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:10:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:10:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:10:24 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:10:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 13:10:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:10:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:10:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:10:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:10:45 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/08 13:10:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:10:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:10:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1354
2021/12/08 13:11:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3854
2021/12/08 13:11:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:11:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:11:04 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/08 13:11:04 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:11:34 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/08 13:11:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 13:11:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:11:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:11:50 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:11:55 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/08 13:11:55 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/08 13:11:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:12:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4921
2021/12/08 13:12:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4921
2021/12/08 13:12:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:12:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:12:13 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/08 13:12:13 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:12:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 13:12:28 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 13:12:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/08 13:12:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/08 13:12:33 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:12:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/08 13:12:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:12:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:12:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:12:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.36s)
=== RUN   TestCPPWorkerCleanup
2021/12/08 13:12:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 13:12:56 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/08 13:12:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:12:56 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/08 13:12:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:13:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:13:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:13:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:13:09 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:13:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/08 13:13:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:13:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:13:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:13:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:13:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:13:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4115
2021/12/08 13:13:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4115
2021/12/08 13:13:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:13:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:13:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:13:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 13:13:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 13:13:58 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 13:14:03 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/08 13:14:03 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/08 13:14:03 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:14:03 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/08 13:14:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:14:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:14:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 13:14:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:14:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:14:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4574
2021/12/08 13:14:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4574
2021/12/08 13:14:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.70s)
=== RUN   TestWithUserXattrs
2021/12/08 13:15:16 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 13:15:16 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/08 13:15:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:15:16 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/08 13:15:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:15:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:15:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:15:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:15:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:15:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:15:33 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:15:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/08 13:15:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 13:15:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 13:15:49 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 13:15:54 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/08 13:15:54 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/08 13:15:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:16:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:16:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4125
2021/12/08 13:16:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4125
2021/12/08 13:16:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:16:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:16:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:16:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 13:16:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 13:16:29 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 13:16:34 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/08 13:16:34 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/08 13:16:34 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:16:34 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/08 13:16:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:16:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:16:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:16:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:16:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:16:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.05s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/08 13:17:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:17:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 13:17:07 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:17:07 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/08 13:17:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:17:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:17:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:17:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:17:20 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:17:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/08 13:17:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 13:17:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 13:17:36 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 13:17:41 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/08 13:17:41 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/08 13:17:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:17:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:17:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:17:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:17:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:17:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:18:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3422
2021/12/08 13:18:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 922
2021/12/08 13:18:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:18:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:18:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:18:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:18:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:18:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15176
2021/12/08 13:18:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15176
2021/12/08 13:18:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/08 13:18:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:18:56 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:19:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:19:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:19:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:19:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:19:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 13:19:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:19:21 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/08 13:19:21 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/08 13:19:21 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:19:21 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/08 13:19:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:19:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/08 13:19:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 13:19:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:19:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:19:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 13:19:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/08 13:19:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.55s)
=== RUN   TestMetaBucketDelete
2021/12/08 13:20:04 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 13:20:04 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 13:20:04 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:20:04 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/08 13:20:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:20:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:20:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:20:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:20:18 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:20:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 13:20:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 13:20:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 13:20:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 13:20:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/08 13:20:39 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/08 13:20:39 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/08 13:20:40 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/08 13:20:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 13:20:45 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 13:21:00 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/08 13:21:00 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/08 13:21:00 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:21:00 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/08 13:21:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:21:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:21:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:21:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.36s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/08 13:21:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:21:23 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:21:23 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:21:23 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 13:21:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:21:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:21:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:21:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:21:36 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:21:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 13:21:56 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/08 13:21:57 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/08 13:22:12 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:22:12 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:22:27 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:22:27 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:22:27 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:22:27 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 13:22:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:22:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:22:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:22:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.14s)
=== RUN   TestSourceBucketDelete
2021/12/08 13:22:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 13:22:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 13:22:50 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:22:50 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/08 13:22:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:22:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:23:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:23:03 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:23:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/08 13:23:09 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 13:23:14 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 13:23:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 13:23:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/08 13:23:24 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/08 13:23:24 handler_test.go:945: Deleting source bucket: default
2021/12/08 13:23:26 handler_test.go:947: Deleted source bucket: default
2021/12/08 13:23:31 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 13:23:31 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 13:23:46 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/08 13:23:46 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/08 13:23:46 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:23:46 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/08 13:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:23:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:23:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:23:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.77s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/08 13:24:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:24:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:24:09 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:24:09 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 13:24:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:24:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:24:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:24:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:24:22 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:24:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 13:24:42 handler_test.go:966: Deleting source bucket: default
2021/12/08 13:24:43 handler_test.go:968: Deleted source bucket: default
2021/12/08 13:24:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:24:58 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:25:13 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:25:13 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:25:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:25:13 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 13:25:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:25:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:25:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:25:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.69s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/08 13:25:36 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:25:36 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:25:36 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:25:36 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/08 13:25:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:25:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:25:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:25:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:25:49 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:25:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 13:26:09 handler_test.go:988: Deleting source bucket: default
2021/12/08 13:26:11 handler_test.go:990: Deleted source bucket: default
2021/12/08 13:26:11 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/08 13:26:11 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/08 13:26:26 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:26:26 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:26:41 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/08 13:26:41 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/08 13:26:41 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:26:42 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/08 13:26:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:26:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:26:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:26:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.10s)
=== RUN   TestUndeployDuringBootstrap
2021/12/08 13:27:04 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:27:04 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 13:27:04 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:27:04 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/08 13:27:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:27:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:27:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:27:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:27:18 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:27:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 13:27:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:27:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:27:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:27:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/08 13:27:31 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/08 13:27:31 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/08 13:27:36 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/08 13:27:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:27:46 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:27:51 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:27:56 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:27:56 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 13:27:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:28:21 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/08 13:28:21 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/08 13:28:21 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:28:21 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/08 13:28:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:28:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:28:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:28:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:28:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:28:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4700
2021/12/08 13:28:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2200
2021/12/08 13:28:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.35s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/08 13:28:59 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:29:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/08 13:29:05 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 13:29:10 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 13:29:15 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 13:29:20 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/08 13:29:20 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/08 13:29:20 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/08 13:29:20 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:29:20 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/08 13:29:25 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/08 13:29:25 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/08 13:29:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:29:25 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/08 13:29:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:29:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:29:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:29:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.69s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/08 13:29:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:29:49 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 13:29:49 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:29:49 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/08 13:29:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:29:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:29:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:30:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:30:02 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:30:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/08 13:30:08 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 13:30:13 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 13:30:18 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 13:30:23 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/08 13:30:23 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/08 13:30:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:30:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:30:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:31:03 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:31:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:31:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:31:13 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 13:31:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:32:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/08 13:32:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/08 13:32:58 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:32:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/08 13:33:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:33:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 13:33:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 13:33:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:33:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:33:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:33:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.04s)
=== RUN   TestUndeployWithKVFailover
2021/12/08 13:33:37 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 13:33:37 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/08 13:33:37 eventingSetup.go:46: No eventing-con process running
2021/12/08 13:33:37 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/08 13:33:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:33:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:33:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:33:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:33:50 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:33:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/08 13:33:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 13:33:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:34:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2496
2021/12/08 13:34:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2496
2021/12/08 13:34:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:34:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 13:34:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 13:34:38 Rebalance progress: 29.16666666666666
2021/12/08 13:34:43 Rebalance progress: 29.16666666666666
2021/12/08 13:34:48 Rebalance progress: 100
2021/12/08 13:34:48 Failing over: [127.0.0.1:19003]
2021/12/08 13:34:58 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/08 13:35:58 Removing node(s): [] from the cluster
2021/12/08 13:36:03 Rebalance progress: 29.16666666666666
2021/12/08 13:36:08 Rebalance progress: 29.16666666666666
2021/12/08 13:36:13 Rebalance progress: 100
2021/12/08 13:36:18 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 13:36:18 App: TestUndeployWithKVFailover got un-deployed
2021/12/08 13:36:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:36:23 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/08 13:36:23 App: TestUndeployWithKVFailover got un-deployed
2021/12/08 13:36:23 No eventing-con process running
2021/12/08 13:36:24 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/08 13:36:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:36:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:36:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.93s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/08 13:36:56 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 13:36:56 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/08 13:36:56 No eventing-con process running
2021/12/08 13:36:57 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/08 13:36:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:37:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:37:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:37:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:37:10 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:37:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/08 13:37:16 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 13:37:21 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 13:37:26 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 13:37:31 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/08 13:37:31 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/08 13:37:31 Starting bucket ops to source bucket
2021/12/08 13:37:51 Waiting for dst bucket item count to get to: 20000 curr count: 9435
2021/12/08 13:37:51 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/08 13:37:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:37:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:38:04 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/08 13:38:09 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:38:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:38:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:38:20 Waiting for dst bucket item count to get to: 0 curr count: 16676
2021/12/08 13:38:26 Waiting for dst bucket item count to get to: 0 curr count: 16676
2021/12/08 13:38:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:38:26 Starting bucket ops to source bucket
2021/12/08 13:38:38 Waiting for dst bucket item count to get to: 10000 curr count: 3433
2021/12/08 13:38:43 Waiting for dst bucket item count to get to: 10000 curr count: 3433
2021/12/08 13:38:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 13:38:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:38:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:38:48 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 13:38:53 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 13:38:58 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 13:39:03 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/08 13:39:03 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/08 13:39:03 No eventing-con process running
2021/12/08 13:39:04 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/08 13:39:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:39:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:39:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:39:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:39:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.06s)
=== RUN   TestTimerOverwrite
2021/12/08 13:39:37 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 13:39:37 App: TestTimerOverwrite got un-deployed
2021/12/08 13:39:37 No eventing-con process running
2021/12/08 13:39:37 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/08 13:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:39:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:39:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:39:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:39:50 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:39:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/08 13:39:56 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 13:40:01 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 13:40:06 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 13:40:11 Waiting for app: TestTimerOverwrite to get deployed
2021/12/08 13:40:11 App: TestTimerOverwrite got deployed
2021/12/08 13:40:21 Starting bucket ops to source bucket
2021/12/08 13:40:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:40:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:40:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:40:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:40:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:40:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 13:41:30 Waiting for dst bucket item count to get to: 5000 curr count: 3989
2021/12/08 13:41:35 Waiting for dst bucket item count to get to: 5000 curr count: 3989
2021/12/08 13:41:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:41:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:41:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:41:50 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 13:41:55 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 13:42:00 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 13:42:05 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/08 13:42:05 App: TestTimerOverwrite got un-deployed
2021/12/08 13:42:06 No eventing-con process running
2021/12/08 13:42:06 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/08 13:42:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:42:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:42:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:42:29 Waiting for dst bucket item count to get to: 0 curr count: 4560
2021/12/08 13:42:34 Waiting for dst bucket item count to get to: 0 curr count: 2020
2021/12/08 13:42:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (187.01s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/08 13:42:39 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 13:42:39 Removing node(s): [] from the cluster
2021/12/08 13:42:44 Rebalance progress: 100
2021/12/08 13:42:44 Failing over: [127.0.0.1:19003]
2021/12/08 13:42:49 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 13:42:49 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 13:42:49 No eventing-con process running
2021/12/08 13:42:49 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/08 13:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:42:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:43:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:43:02 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:43:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/08 13:43:03 Starting bucket ops to source bucket
2021/12/08 13:43:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:43:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:43:13 Context: Post to set retry counter, response code: 200 dump: 
2021/12/08 13:43:13 set retry counter, response &{[] }
2021/12/08 13:44:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 13:44:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 13:44:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:44:19 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/08 13:44:24 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/08 13:44:24 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/08 13:44:24 No eventing-con process running
2021/12/08 13:44:24 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/08 13:44:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:44:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:44:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:44:47 Removing node(s): [] from the cluster
2021/12/08 13:44:52 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.07s)
=== RUN   TestOnUpdateSrcMutation
2021/12/08 13:45:02 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 13:45:02 App: TestOnUpdateSrcMutation got un-deployed
2021/12/08 13:45:02 No eventing-con process running
2021/12/08 13:45:02 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/08 13:45:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:45:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:45:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:45:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:45:15 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:45:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/08 13:45:16 Starting bucket ops to source bucket
2021/12/08 13:45:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 13:45:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:45:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:45:48 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 13:45:53 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 13:45:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 13:46:03 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/08 13:46:03 App: TestOnUpdateSrcMutation got un-deployed
2021/12/08 13:46:03 No eventing-con process running
2021/12/08 13:46:03 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/08 13:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:46:14 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:46:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:46:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:46:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.51s)
=== RUN   TestOnDeleteSrcMutation
2021/12/08 13:46:31 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 13:46:31 App: TestOnDeleteSrcMutation got un-deployed
2021/12/08 13:46:31 No eventing-con process running
2021/12/08 13:46:31 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/08 13:46:32 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:46:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:46:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:46:44 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:46:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/08 13:46:46 Starting bucket ops to source bucket
2021/12/08 13:47:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:47:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:47:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:47:19 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 13:47:24 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 13:47:29 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 13:47:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/08 13:47:34 App: TestOnDeleteSrcMutation got un-deployed
2021/12/08 13:47:34 No eventing-con process running
2021/12/08 13:47:34 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/08 13:47:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:47:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:47:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.34s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/08 13:48:02 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 13:48:02 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/08 13:48:02 No eventing-con process running
2021/12/08 13:48:02 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/08 13:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:48:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:48:16 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:48:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 13:48:17 Starting bucket ops to source bucket
2021/12/08 13:49:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 13:49:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:49:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:49:09 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 13:49:14 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 13:49:19 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 13:49:24 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/08 13:49:24 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/08 13:49:24 No eventing-con process running
2021/12/08 13:49:24 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/08 13:49:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:49:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 13:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:49:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:49:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.51s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/08 13:49:52 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 13:49:52 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/08 13:49:52 No eventing-con process running
2021/12/08 13:49:52 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/08 13:49:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:49:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:50:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:50:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:50:05 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:50:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/08 13:50:06 Starting bucket ops to source bucket
2021/12/08 13:50:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 13:50:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 13:50:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:50:50 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 13:50:55 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 13:51:00 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 13:51:05 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/08 13:51:05 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/08 13:51:05 No eventing-con process running
2021/12/08 13:51:05 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/08 13:51:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:51:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 13:51:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:51:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:51:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.35s)
=== RUN   TestInterHandlerRecursion
2021/12/08 13:51:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:51:33 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 13:51:33 No eventing-con process running
2021/12/08 13:51:33 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/08 13:51:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:51:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:51:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:51:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:51:51 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:51:51 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 13:51:52 No eventing-con process running
2021/12/08 13:51:52 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 13:51:53 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:51:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:52:00 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:52:05 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:52:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/08 13:52:06 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:52:11 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:52:16 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:52:21 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:52:26 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:52:26 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/08 13:52:26 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:52:26 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/08 13:52:26 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/08 13:52:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:52:31 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:52:36 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:52:41 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:52:46 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:52:46 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 13:52:46 No eventing-con process running
2021/12/08 13:52:46 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/08 13:52:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:53:04 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:53:04 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 13:53:04 No eventing-con process running
2021/12/08 13:53:04 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 13:53:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:53:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:53:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.77s)
=== RUN   TestInterBucketRecursion
2021/12/08 13:53:22 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:53:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 13:53:23 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:53:23 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:53:23 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/08 13:53:28 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 13:53:33 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 13:53:38 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 13:53:43 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/08 13:53:43 App: TestInterBucketRecursion_function1 got deployed
2021/12/08 13:53:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:53:48 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:53:53 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:53:58 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:54:03 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:54:03 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/08 13:54:03 No eventing-con process running
2021/12/08 13:54:04 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/08 13:54:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:54:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:54:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:54:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:54:22 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/08 13:54:22 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/08 13:54:22 No eventing-con process running
2021/12/08 13:54:22 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/08 13:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:54:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.58s)
=== RUN   TestLargeHandler
2021/12/08 13:54:35 Changed force_compress value to false
2021/12/08 13:54:35 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:54:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/08 13:54:35 Changed force_compress value to true
2021/12/08 13:54:35 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:54:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/08 13:54:42 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 13:54:47 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 13:54:52 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 13:54:57 Waiting for app: TestLargeHandler_function to get deployed
2021/12/08 13:54:57 App: TestLargeHandler_function got deployed
2021/12/08 13:54:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:55:02 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 13:55:07 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 13:55:12 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 13:55:17 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/08 13:55:17 App: TestLargeHandler_function got un-deployed
2021/12/08 13:55:17 No eventing-con process running
2021/12/08 13:55:17 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/08 13:55:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:55:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:55:25 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:55:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.31s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/08 13:55:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:55:40 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 13:55:40 No eventing-con process running
2021/12/08 13:55:40 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/08 13:55:41 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:55:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:55:48 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:55:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:55:58 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:55:58 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 13:55:58 No eventing-con process running
2021/12/08 13:55:58 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/08 13:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:56:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:56:06 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:56:11 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:56:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/08 13:56:12 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:56:17 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:56:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:56:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:56:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/08 13:56:32 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/08 13:56:33 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:56:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/08 13:56:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:56:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 13:56:44 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 13:56:49 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 13:56:54 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/08 13:56:54 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/08 13:56:54 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/08 13:56:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:56:59 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:57:04 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:57:09 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:57:14 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/08 13:57:14 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/08 13:57:14 No eventing-con process running
2021/12/08 13:57:14 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/08 13:57:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:57:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:57:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:57:33 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:57:38 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:57:43 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:57:48 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/08 13:57:48 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/08 13:57:48 No eventing-con process running
2021/12/08 13:57:48 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/08 13:57:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:57:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:58:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.27s)
=== RUN   TestAllowInterBucketRecursion
2021/12/08 13:58:06 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:58:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 13:58:07 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:58:12 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 13:58:17 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 13:58:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 13:58:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/08 13:58:27 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/08 13:58:27 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 13:58:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 13:58:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/08 13:58:34 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 13:58:39 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 13:58:44 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 13:58:49 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/08 13:58:49 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/08 13:58:49 TestAllowInterBucketRecursion_function2 is deployed
2021/12/08 13:58:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:58:54 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:58:59 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:59:04 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:59:09 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/08 13:59:09 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/08 13:59:09 No eventing-con process running
2021/12/08 13:59:09 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/08 13:59:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:59:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 13:59:28 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 13:59:33 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 13:59:38 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 13:59:43 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/08 13:59:43 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/08 13:59:43 No eventing-con process running
2021/12/08 13:59:43 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/08 13:59:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:59:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 13:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/08 13:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.10s)
=== RUN   TestInterBucketRecursion2
2021/12/08 14:00:02 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:00:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:00:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:00:25 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:00:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 14:00:32 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 14:00:37 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 14:00:42 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 14:00:47 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/08 14:00:47 App: TestInterBucketRecursion2_function1 got deployed
2021/12/08 14:00:47 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:00:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 14:00:53 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 14:00:58 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 14:01:03 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 14:01:08 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/08 14:01:08 App: TestInterBucketRecursion2_function2 got deployed
2021/12/08 14:01:08 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:01:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 14:01:14 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 14:01:19 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 14:01:24 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 14:01:29 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/08 14:01:29 App: TestInterBucketRecursion2_function3 got deployed
2021/12/08 14:01:29 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:01:29 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/08 14:01:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:01:34 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 14:01:39 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 14:01:44 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 14:01:49 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/08 14:01:49 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/08 14:01:49 No eventing-con process running
2021/12/08 14:01:49 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/08 14:01:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:02:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:02:07 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 14:02:12 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 14:02:17 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 14:02:22 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/08 14:02:22 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/08 14:02:22 No eventing-con process running
2021/12/08 14:02:23 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/08 14:02:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:02:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:02:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:02:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:02:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:02:41 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 14:02:46 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 14:02:51 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 14:02:56 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/08 14:02:56 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/08 14:02:56 No eventing-con process running
2021/12/08 14:02:56 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/08 14:02:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:03:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:03:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:03:14 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/08 14:03:14 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/08 14:03:14 No eventing-con process running
2021/12/08 14:03:14 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/08 14:03:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:03:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:03:22 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (212.91s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/08 14:03:34 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 14:03:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/08 14:03:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:03:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:04:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:04:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:04:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:04:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:05:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:05:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:05:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/08 14:05:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:40} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/08 14:06:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:06:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 14:06:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 14:06:18 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/08 14:06:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:06:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 14:06:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 14:06:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/08 14:06:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:06:28 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/08 14:06:28 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/08 14:06:28 eventingSetup.go:46: No eventing-con process running
2021/12/08 14:06:28 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/08 14:06:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:06:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:06:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:06:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:06:46 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:06:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 14:06:53 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 14:06:58 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 14:07:03 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 14:07:08 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/08 14:07:08 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 50ec84f50f20f053bef5ba2d7a27244e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 3251124449281975ff2a3ddd4f098ca1
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: c64ffcd42db24688dd2d82b8fefed45a
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]


2021/12/08 14:07:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:07:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 14:07:13 Rebalance progress: 29.16666666666666
2021/12/08 14:07:18 Rebalance progress: 29.16666666666666
2021/12/08 14:07:23 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 50ec84f50f20f053bef5ba2d7a27244e
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: 3251124449281975ff2a3ddd4f098ca1
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: c64ffcd42db24688dd2d82b8fefed45a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/08 14:07:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:07:23 handler_test.go:1674: Deleting source bucket: default
2021/12/08 14:07:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 14:07:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 14:07:30 Rebalance progress: 29.16666666666666
2021/12/08 14:07:35 Rebalance progress: 29.16666666666666
2021/12/08 14:07:40 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 904
	vb list dump: [0-24, 26-29, 31-41, 43-57, 59, 61-70, 72-78, 81, 83-85, 87-88, 91-112, 114-117, 119-141, 145-153, 156-170, 172-182, 184-187, 189-196, 198-202, 204-214, 216-217, 219-220, 222-240, 242-257, 259-263, 265-268, 270-284, 286, 289-291, 293-304, 306-308, 310-311, 313-315, 317-326, 328-340, 342-346, 348-356, 359-368, 371-376, 378-394, 396-404, 406, 408-415, 417-420, 422-424, 426-440, 442-456, 458-461, 463-465, 467-470, 472, 474, 477-479, 481, 483-484, 486-498, 501, 504-575, 577-581, 583-593, 595-606, 608-614, 616-619, 621-624, 628-630, 632-643, 645-668, 670-682, 684-688, 690-702, 704-705, 707-716, 718-729, 732-735, 737-740, 742-745, 747-748, 750-764, 766-768, 770-778, 780-784, 786-832, 835-843, 845-853, 855-859, 862-870, 872-874, 876-878, 880-887, 889-892, 894-904, 906-908, 910-925, 927-948, 950-963, 965, 968-979, 981-986, 988-994, 996-1002, 1004-1013, 1015-1019, 1021-1022]
	stream status: stopped
	len: 120
	vb list dump: [25, 30, 42, 58, 60, 71, 79-80, 82, 86, 89-90, 113, 118, 142-144, 154-155, 171, 183, 188, 197, 203, 215, 218, 221, 241, 258, 264, 269, 285, 287-288, 292, 305, 309, 312, 316, 327, 341, 347, 357-358, 369-370, 377, 395, 405, 407, 416, 421, 425, 441, 457, 462, 466, 471, 473, 475-476, 480, 482, 485, 499-500, 502-503, 576, 582, 594, 607, 615, 620, 625-627, 631, 644, 669, 683, 689, 703, 706, 717, 730-731, 736, 741, 746, 749, 765, 769, 779, 785, 833-834, 844, 854, 860-861, 871, 875, 879, 888, 893, 905, 909, 926, 949, 964, 966-967, 980, 987, 995, 1003, 1014, 1020, 1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 50ec84f50f20f053bef5ba2d7a27244e
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-24, 26-29, 31-41, 43-57, 59, 61-70, 72-78, 81, 83-85, 87-88, 91-112]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-117, 119-141, 145-153, 156-170, 172-182, 184-187, 189-196, 198-202, 204-214, 216-217, 219-220, 222-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [228-240, 242-257, 259-263, 265-268, 270-284, 286, 289-291, 293-304, 306-308, 310-311, 313-315, 317-326, 328-340]
Producer node: 172.31.2.134:9300	Node UUID: 3251124449281975ff2a3ddd4f098ca1
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-925, 927-948, 950-963, 965, 968-979, 981-986, 988-994, 996-1002, 1004-1013, 1015-1019, 1021-1022]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [684-688, 690-702, 704-705, 707-716, 718-729, 732-735, 737-740, 742-745, 747-748, 750-764, 766-768, 770-778, 780-784, 786-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-832, 835-843, 845-853, 855-859, 862-870, 872-874, 876-878, 880-887, 889-892, 894-904, 906-908, 910]
Producer node: 127.0.0.1:9303	Node UUID: c64ffcd42db24688dd2d82b8fefed45a
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [456, 458-461, 463-465, 467-470, 472, 474, 477-479, 481, 483-484, 486-498, 501, 504-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-575, 577-581, 583-593, 595-606, 608-614, 616-619, 621-624, 628-630, 632-643, 645-668, 670-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-346, 348-356, 359-368, 371-376, 378-394, 396-404, 406, 408-415, 417-420, 422-424, 426-440, 442-455]


2021/12/08 14:07:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:07:45 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/08 14:07:45 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/08 14:07:50 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/08 14:07:50 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/08 14:07:50 eventingSetup.go:46: No eventing-con process running
2021/12/08 14:07:50 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/08 14:07:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:07:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:07:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:08:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:08:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 14:08:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 14:08:09 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (281.98s)
=== RUN   TestN1QLRecursion
2021/12/08 14:08:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 14:08:12 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 14:08:12 Removing node(s): [] from the cluster
2021/12/08 14:08:17 Rebalance progress: 100
2021/12/08 14:08:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:08:29 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:08:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:08:40 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:08:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 14:08:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 14:08:46 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:08:51 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:08:56 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:01 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:06 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:11 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:16 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:21 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/08 14:09:21 App: TestN1QLRecursion_function1 got deployed
2021/12/08 14:09:21 TestN1QLRecursion_function1 is deployed
2021/12/08 14:09:21 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:09:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 14:09:22 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/08 14:09:27 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 14:09:32 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 14:09:37 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 14:09:42 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/08 14:09:42 App: TestN1QLRecursion_function3 got deployed
2021/12/08 14:09:42 TestN1QLRecursion_function3 is deployed
2021/12/08 14:09:42 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:09:42 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/08 14:09:42 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/08 14:09:42 Success: TestN1QLRecursion_function2 is not deployed
2021/12/08 14:09:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:09:47 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 14:09:52 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 14:09:57 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 14:10:02 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/08 14:10:02 App: TestN1QLRecursion_function1 got un-deployed
2021/12/08 14:10:02 No eventing-con process running
2021/12/08 14:10:02 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/08 14:10:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:10:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:10:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:10:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:10:21 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 14:10:26 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 14:10:31 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 14:10:36 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/08 14:10:36 App: TestN1QLRecursion_function3 got un-deployed
2021/12/08 14:10:36 No eventing-con process running
2021/12/08 14:10:36 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/08 14:10:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:10:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:10:45 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:10:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:10:52 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 14:10:57 Rebalance progress: 100
--- PASS: TestN1QLRecursion (165.38s)
=== RUN   TestN1QLAllowRecursion
2021/12/08 14:10:57 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 14:10:57 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 14:10:57 Removing node(s): [] from the cluster
2021/12/08 14:11:02 Rebalance progress: 100
2021/12/08 14:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:11:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:11:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:11:24 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:11:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/08 14:11:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/08 14:11:31 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 14:11:36 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 14:11:41 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 14:11:46 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/08 14:11:46 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/08 14:11:46 TestN1QLAllowRecursion_function1 is deployed
2021/12/08 14:11:46 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:11:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/08 14:11:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/08 14:11:52 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 14:11:57 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 14:12:02 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 14:12:07 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/08 14:12:07 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/08 14:12:07 TestN1QLAllowRecursion_function3 is deployed
2021/12/08 14:12:07 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:12:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/08 14:12:08 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/08 14:12:13 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 14:12:18 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 14:12:23 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 14:12:28 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/08 14:12:28 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/08 14:12:28 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/08 14:12:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:12:33 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 14:12:38 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 14:12:43 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 14:12:48 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/08 14:12:48 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/08 14:12:48 No eventing-con process running
2021/12/08 14:12:48 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/08 14:12:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:12:57 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:13:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:13:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:13:07 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 14:13:12 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 14:13:17 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 14:13:22 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/08 14:13:22 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/08 14:13:22 No eventing-con process running
2021/12/08 14:13:22 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/08 14:13:24 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:13:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:13:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:13:41 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 14:13:46 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 14:13:51 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 14:13:56 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/08 14:13:56 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/08 14:13:56 No eventing-con process running
2021/12/08 14:13:56 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/08 14:13:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:14:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:14:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:14:11 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 14:14:16 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.81s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/08 14:14:16 Sleeping for some time
2021/12/08 14:14:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 14:14:26 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/08 14:14:26 No eventing-con process running
2021/12/08 14:14:26 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/08 14:14:28 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:14:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:14:35 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:14:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:14:40 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:14:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 14:14:41 Starting bucket ops to source bucket
2021/12/08 14:15:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 14:15:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 14:15:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 14:15:35 Starting bucket ops to source bucket
2021/12/08 14:15:36 Starting bucket ops to source bucket
2021/12/08 14:16:42 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/08 14:16:47 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/08 14:16:47 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/08 14:16:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:16:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:16:52 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 14:16:57 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 14:17:02 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 14:17:07 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/08 14:17:07 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/08 14:17:07 No eventing-con process running
2021/12/08 14:17:07 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/08 14:17:09 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:17:19 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:17:24 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/08 14:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:17:26 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:17:36 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/08 14:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.69s)
=== RUN   TestTimerOverWriteSameReference
2021/12/08 14:17:41 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 14:17:41 App: TestTimerOverWriteSameReference got un-deployed
2021/12/08 14:17:41 No eventing-con process running
2021/12/08 14:17:41 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/08 14:17:43 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:17:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:17:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:17:55 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:17:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/08 14:18:01 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 14:18:06 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 14:18:11 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 14:18:16 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/08 14:18:16 App: TestTimerOverWriteSameReference got deployed
2021/12/08 14:18:26 Starting bucket ops to source bucket
2021/12/08 14:19:27 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/08 14:19:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:19:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:19:42 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 14:19:47 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 14:19:52 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 14:19:57 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/08 14:19:57 App: TestTimerOverWriteSameReference got un-deployed
2021/12/08 14:19:57 No eventing-con process running
2021/12/08 14:19:57 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/08 14:19:59 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:20:09 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:20:14 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:20:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:20:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:20:26 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/08 14:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.71s)
=== RUN   TestCancelTimerBucketop
2021/12/08 14:20:31 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 14:20:31 App: TestCancelTimerBucketop got un-deployed
2021/12/08 14:20:31 No eventing-con process running
2021/12/08 14:20:31 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/08 14:20:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:20:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:20:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:20:44 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:20:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/08 14:20:51 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 14:20:56 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 14:21:01 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 14:21:06 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/08 14:21:06 App: TestCancelTimerBucketop got deployed
2021/12/08 14:21:06 Starting bucket ops to source bucket
2021/12/08 14:21:13 Starting bucket ops to source bucket
2021/12/08 14:22:15 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 14:22:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:22:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:22:20 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 14:22:25 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 14:22:30 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 14:22:35 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/08 14:22:35 App: TestCancelTimerBucketop got un-deployed
2021/12/08 14:22:35 No eventing-con process running
2021/12/08 14:22:35 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/08 14:22:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:22:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:22:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:22:59 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:23:04 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:23:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.26s)
=== RUN   TestJSExpiryDate
2021/12/08 14:23:09 Starting bucket ops to source bucket
2021/12/08 14:23:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 14:23:22 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/08 14:23:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/08 14:23:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:23:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:23:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:23:58 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 14:24:03 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 14:24:08 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 14:24:14 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/08 14:24:14 App: TestJSExpiryDate got un-deployed
2021/12/08 14:24:14 No eventing-con process running
2021/12/08 14:24:14 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/08 14:24:15 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:24:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:24:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.61s)
=== RUN   TestBinaryDoc
2021/12/08 14:24:33 Starting bucket ops to source bucket
2021/12/08 14:24:33 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:24:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/08 14:24:34 Starting bucket ops to source bucket
2021/12/08 14:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:24:34 Deleting source bucket docs now
2021/12/08 14:24:34 Starting bucket ops to source bucket
2021/12/08 14:25:11 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/08 14:25:16 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/08 14:25:16 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/08 14:25:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:25:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:25:21 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 14:25:26 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 14:25:31 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 14:25:36 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/08 14:25:36 App: TestBinaryDoc got un-deployed
2021/12/08 14:25:36 No eventing-con process running
2021/12/08 14:25:37 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/08 14:25:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:25:48 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/08 14:25:53 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/08 14:25:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:25:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:26:05 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/08 14:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.37s)
=== RUN   TestConstantBindings
2021/12/08 14:26:05 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:26:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/08 14:26:06 Starting bucket ops to source bucket
2021/12/08 14:26:12 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 14:26:17 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 14:26:22 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/08 14:26:27 Waiting for dst bucket item count to get to: 6000 curr count: 1719
2021/12/08 14:26:32 Waiting for dst bucket item count to get to: 6000 curr count: 1719
2021/12/08 14:26:32 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/08 14:26:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:26:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:26:37 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 14:26:42 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 14:26:47 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 14:26:52 Waiting for app: TestConstantBindings to get un-deployed
2021/12/08 14:26:52 App: TestConstantBindings got un-deployed
2021/12/08 14:26:52 No eventing-con process running
2021/12/08 14:26:52 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/08 14:26:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:27:04 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:27:09 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:27:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:27:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:27:21 Waiting for dst bucket item count to get to: 0 curr count: 5303
2021/12/08 14:27:26 Waiting for dst bucket item count to get to: 0 curr count: 5303
2021/12/08 14:27:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.67s)
=== RUN   TestCodeInsight
2021/12/08 14:27:31 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 14:27:31 App: TestCodeInsight got un-deployed
2021/12/08 14:27:31 No eventing-con process running
2021/12/08 14:27:31 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/08 14:27:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:27:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:27:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:27:44 Starting bucket ops to source bucket
2021/12/08 14:27:45 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:27:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/08 14:27:51 Waiting for app: TestCodeInsight to get deployed
2021/12/08 14:27:56 Waiting for app: TestCodeInsight to get deployed
2021/12/08 14:28:01 Waiting for app: TestCodeInsight to get deployed
2021/12/08 14:28:06 Waiting for app: TestCodeInsight to get deployed
2021/12/08 14:28:06 App: TestCodeInsight got deployed
2021/12/08 14:28:11 Waiting for dst bucket item count to get to: 1000 curr count: 320
2021/12/08 14:28:11 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 14:28:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:28:16 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 14:28:22 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 14:28:27 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 14:28:32 Waiting for app: TestCodeInsight to get un-deployed
2021/12/08 14:28:32 App: TestCodeInsight got un-deployed
2021/12/08 14:28:32 No eventing-con process running
2021/12/08 14:28:32 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/08 14:28:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:28:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:28:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:28:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:28:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:29:00 Waiting for dst bucket item count to get to: 0 curr count: 707
2021/12/08 14:29:05 Waiting for dst bucket item count to get to: 0 curr count: 707
2021/12/08 14:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.42s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12736.878s

Functional tests took 3 hours, 32 minutes and 20 seconds.

N1ql tests

2021/12/08 14:29:15 Started n1ql
2021/12/08 14:29:24 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/08 14:29:25 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/08 14:29:25 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 14:29:25 Removing node(s): [] from the cluster
2021/12/08 14:29:30 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/08 14:29:35 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/08 14:29:35 App: TestRecursiveMutationN1QL got un-deployed
2021/12/08 14:29:35 No eventing-con process running
2021/12/08 14:29:35 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/08 14:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:29:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:29:44 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:29:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:29:49 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:29:49 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.72s)
=== RUN   TestFlexReset1
2021/12/08 14:29:54 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 14:29:54 App: TestFlexReset1 got un-deployed
2021/12/08 14:29:54 No eventing-con process running
2021/12/08 14:29:54 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/08 14:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:30:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:30:03 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:30:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:30:08 Starting bucket ops to source bucket
2021/12/08 14:30:08 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:30:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/08 14:30:14 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 14:30:19 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 14:30:24 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 14:30:29 Waiting for app: TestFlexReset1 to get deployed
2021/12/08 14:30:29 App: TestFlexReset1 got deployed
2021/12/08 14:30:34 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/08 14:30:34 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 14:30:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:30:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:30:40 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 14:30:45 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 14:30:50 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 14:30:55 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/08 14:30:55 App: TestFlexReset1 got un-deployed
2021/12/08 14:30:55 No eventing-con process running
2021/12/08 14:30:55 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/08 14:30:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:31:06 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:31:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:31:18 Waiting for dst bucket item count to get to: 0 curr count: 751
2021/12/08 14:31:23 Waiting for dst bucket item count to get to: 0 curr count: 251
2021/12/08 14:31:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.37s)
=== RUN   TestFlexReset2
2021/12/08 14:31:28 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 14:31:28 App: TestFlexReset2 got un-deployed
2021/12/08 14:31:28 No eventing-con process running
2021/12/08 14:31:28 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/08 14:31:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:31:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:31:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:31:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:31:42 Starting bucket ops to source bucket
2021/12/08 14:31:42 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:31:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/08 14:31:49 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 14:31:54 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 14:31:59 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 14:32:04 Waiting for app: TestFlexReset2 to get deployed
2021/12/08 14:32:04 App: TestFlexReset2 got deployed
2021/12/08 14:32:09 Waiting for dst bucket item count to get to: 1000 curr count: 728
2021/12/08 14:32:14 Waiting for dst bucket item count to get to: 1000 curr count: 728
2021/12/08 14:32:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/08 14:32:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:32:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:32:19 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 14:32:24 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 14:32:29 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 14:32:34 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/08 14:32:34 App: TestFlexReset2 got un-deployed
2021/12/08 14:32:34 No eventing-con process running
2021/12/08 14:32:34 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/08 14:32:36 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:32:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/08 14:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:32:57 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.20s)
=== RUN   TestN1QLLabelledBreak
2021/12/08 14:33:02 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 14:33:02 App: TestN1QLLabelledBreak got un-deployed
2021/12/08 14:33:02 No eventing-con process running
2021/12/08 14:33:02 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/08 14:33:04 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:33:16 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:33:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/08 14:33:22 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 14:33:27 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 14:33:32 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 14:33:37 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/08 14:33:37 App: TestN1QLLabelledBreak got deployed
2021/12/08 14:33:37 Starting bucket ops to source bucket
2021/12/08 14:33:46 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 14:33:51 Waiting for dst bucket item count to get to: 10000 curr count: 142
2021/12/08 14:33:57 Waiting for dst bucket item count to get to: 10000 curr count: 142
2021/12/08 14:34:02 Waiting for dst bucket item count to get to: 10000 curr count: 3326
2021/12/08 14:34:07 Waiting for dst bucket item count to get to: 10000 curr count: 3326
2021/12/08 14:34:12 Waiting for dst bucket item count to get to: 10000 curr count: 7347
2021/12/08 14:34:17 Waiting for dst bucket item count to get to: 10000 curr count: 7347
2021/12/08 14:34:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 14:34:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:34:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:34:22 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 14:34:27 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 14:34:32 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 14:34:37 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/08 14:34:37 App: TestN1QLLabelledBreak got un-deployed
2021/12/08 14:34:37 No eventing-con process running
2021/12/08 14:34:37 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/08 14:34:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:34:49 Waiting for dst bucket item count to get to: 0 curr count: 4287
2021/12/08 14:34:54 Waiting for dst bucket item count to get to: 0 curr count: 1787
2021/12/08 14:34:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:34:55 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:35:05 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 14:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.16s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/08 14:35:11 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 14:35:11 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/08 14:35:11 No eventing-con process running
2021/12/08 14:35:11 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/08 14:35:12 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:35:19 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:35:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:35:24 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:35:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/08 14:35:30 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 14:35:35 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 14:35:40 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 14:35:45 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/08 14:35:45 App: TestN1QLUnlabelledBreak got deployed
2021/12/08 14:35:45 Starting bucket ops to source bucket
2021/12/08 14:35:55 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 14:36:00 Waiting for dst bucket item count to get to: 10000 curr count: 625
2021/12/08 14:36:05 Waiting for dst bucket item count to get to: 10000 curr count: 625
2021/12/08 14:36:10 Waiting for dst bucket item count to get to: 10000 curr count: 5914
2021/12/08 14:36:15 Waiting for dst bucket item count to get to: 10000 curr count: 5914
2021/12/08 14:36:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 14:36:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:36:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:36:20 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 14:36:25 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 14:36:30 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 14:36:35 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/08 14:36:35 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/08 14:36:35 No eventing-con process running
2021/12/08 14:36:35 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/08 14:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:36:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 14:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:36:49 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:36:59 Waiting for dst bucket item count to get to: 0 curr count: 8672
2021/12/08 14:37:04 Waiting for dst bucket item count to get to: 0 curr count: 3672
2021/12/08 14:37:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.04s)
=== RUN   TestN1QLThrowStatement
2021/12/08 14:37:09 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 14:37:09 App: TestN1QLThrowStatement got un-deployed
2021/12/08 14:37:09 No eventing-con process running
2021/12/08 14:37:09 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/08 14:37:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:37:22 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:37:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/08 14:37:28 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 14:37:33 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 14:37:38 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 14:37:43 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/08 14:37:43 App: TestN1QLThrowStatement got deployed
2021/12/08 14:37:43 Starting bucket ops to source bucket
2021/12/08 14:37:53 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/08 14:37:58 Waiting for dst bucket item count to get to: 10000 curr count: 278
2021/12/08 14:38:03 Waiting for dst bucket item count to get to: 10000 curr count: 886
2021/12/08 14:38:08 Waiting for dst bucket item count to get to: 10000 curr count: 2845
2021/12/08 14:38:13 Waiting for dst bucket item count to get to: 10000 curr count: 4920
2021/12/08 14:38:18 Waiting for dst bucket item count to get to: 10000 curr count: 6929
2021/12/08 14:38:23 Waiting for dst bucket item count to get to: 10000 curr count: 8906
2021/12/08 14:38:28 Waiting for dst bucket item count to get to: 10000 curr count: 9660
2021/12/08 14:38:28 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/08 14:38:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:38:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:38:33 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 14:38:38 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 14:38:43 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 14:38:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/08 14:38:48 App: TestN1QLThrowStatement got un-deployed
2021/12/08 14:38:48 No eventing-con process running
2021/12/08 14:38:48 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/08 14:38:50 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:39:00 Waiting for dst bucket item count to get to: 0 curr count: 2626
2021/12/08 14:39:05 Waiting for dst bucket item count to get to: 0 curr count: 2626
2021/12/08 14:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:39:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/08 14:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.06s)
=== RUN   TestN1QLNestedForLoop
2021/12/08 14:39:22 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 14:39:22 App: TestN1QLNestedForLoop got un-deployed
2021/12/08 14:39:22 No eventing-con process running
2021/12/08 14:39:22 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/08 14:39:23 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:39:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:39:30 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:39:35 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:39:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/08 14:39:41 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 14:39:46 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 14:39:51 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 14:39:56 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/08 14:39:56 App: TestN1QLNestedForLoop got deployed
2021/12/08 14:39:56 Starting bucket ops to source bucket
2021/12/08 14:40:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/08 14:40:10 Waiting for dst bucket item count to get to: 5000 curr count: 112
2021/12/08 14:40:15 Waiting for dst bucket item count to get to: 5000 curr count: 112
2021/12/08 14:40:20 Waiting for dst bucket item count to get to: 5000 curr count: 2088
2021/12/08 14:40:25 Waiting for dst bucket item count to get to: 5000 curr count: 2088
2021/12/08 14:40:30 Waiting for dst bucket item count to get to: 5000 curr count: 4471
2021/12/08 14:40:35 Waiting for dst bucket item count to get to: 5000 curr count: 4471
2021/12/08 14:40:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 14:40:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:40:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:40:41 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 14:40:46 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 14:40:51 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 14:40:56 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/08 14:40:56 App: TestN1QLNestedForLoop got un-deployed
2021/12/08 14:40:56 No eventing-con process running
2021/12/08 14:40:56 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/08 14:40:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:41:08 Waiting for dst bucket item count to get to: 0 curr count: 2506
2021/12/08 14:41:13 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/08 14:41:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:41:14 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:41:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 14:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (127.90s)
=== RUN   TestN1QLAutoClose
2021/12/08 14:41:30 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 14:41:30 App: TestN1QLAutoClose got un-deployed
2021/12/08 14:41:30 No eventing-con process running
2021/12/08 14:41:30 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/08 14:41:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:41:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:41:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:41:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:41:43 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:41:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/08 14:41:49 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 14:41:54 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 14:41:59 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 14:42:04 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/08 14:42:04 App: TestN1QLAutoClose got deployed
2021/12/08 14:42:04 Starting bucket ops to source bucket
2021/12/08 14:42:10 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/08 14:42:15 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/08 14:42:20 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/08 14:42:25 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/08 14:42:30 Waiting for dst bucket item count to get to: 500 curr count: 30
2021/12/08 14:42:35 Waiting for dst bucket item count to get to: 500 curr count: 40
2021/12/08 14:42:40 Waiting for dst bucket item count to get to: 500 curr count: 51
2021/12/08 14:42:45 Waiting for dst bucket item count to get to: 500 curr count: 62
2021/12/08 14:42:50 Waiting for dst bucket item count to get to: 500 curr count: 74
2021/12/08 14:42:55 Waiting for dst bucket item count to get to: 500 curr count: 85
2021/12/08 14:43:00 Waiting for dst bucket item count to get to: 500 curr count: 96
2021/12/08 14:43:05 Waiting for dst bucket item count to get to: 500 curr count: 117
2021/12/08 14:43:10 Waiting for dst bucket item count to get to: 500 curr count: 117
2021/12/08 14:43:15 Waiting for dst bucket item count to get to: 500 curr count: 117
2021/12/08 14:43:20 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/08 14:43:25 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/08 14:43:30 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/08 14:43:35 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/08 14:43:40 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/08 14:43:45 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/08 14:43:50 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/08 14:43:55 Waiting for dst bucket item count to get to: 500 curr count: 204
2021/12/08 14:44:00 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/08 14:44:05 Waiting for dst bucket item count to get to: 500 curr count: 226
2021/12/08 14:44:10 Waiting for dst bucket item count to get to: 500 curr count: 247
2021/12/08 14:44:15 Waiting for dst bucket item count to get to: 500 curr count: 247
2021/12/08 14:44:20 Waiting for dst bucket item count to get to: 500 curr count: 267
2021/12/08 14:44:25 Waiting for dst bucket item count to get to: 500 curr count: 267
2021/12/08 14:44:30 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/08 14:44:35 Waiting for dst bucket item count to get to: 500 curr count: 288
2021/12/08 14:44:40 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/08 14:44:45 Waiting for dst bucket item count to get to: 500 curr count: 310
2021/12/08 14:44:50 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/08 14:44:55 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/08 14:45:00 Waiting for dst bucket item count to get to: 500 curr count: 351
2021/12/08 14:45:05 Waiting for dst bucket item count to get to: 500 curr count: 351
2021/12/08 14:45:10 Waiting for dst bucket item count to get to: 500 curr count: 374
2021/12/08 14:45:15 Waiting for dst bucket item count to get to: 500 curr count: 385
2021/12/08 14:45:20 Waiting for dst bucket item count to get to: 500 curr count: 396
2021/12/08 14:45:25 Waiting for dst bucket item count to get to: 500 curr count: 407
2021/12/08 14:45:30 Waiting for dst bucket item count to get to: 500 curr count: 416
2021/12/08 14:45:35 Waiting for dst bucket item count to get to: 500 curr count: 430
2021/12/08 14:45:40 Waiting for dst bucket item count to get to: 500 curr count: 438
2021/12/08 14:45:45 Waiting for dst bucket item count to get to: 500 curr count: 449
2021/12/08 14:45:50 Waiting for dst bucket item count to get to: 500 curr count: 460
2021/12/08 14:45:55 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/08 14:46:00 Waiting for dst bucket item count to get to: 500 curr count: 482
2021/12/08 14:46:05 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/08 14:46:05 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/08 14:46:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:46:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:46:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 14:46:15 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 14:46:21 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 14:46:26 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/08 14:46:26 App: TestN1QLAutoClose got un-deployed
2021/12/08 14:46:26 No eventing-con process running
2021/12/08 14:46:26 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/08 14:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:46:37 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:46:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:46:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/08 14:46:54 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/08 14:46:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.57s)
=== RUN   TestN1QLPosParams
2021/12/08 14:46:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 14:46:59 App: TestN1QLPosParams got un-deployed
2021/12/08 14:46:59 No eventing-con process running
2021/12/08 14:46:59 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/08 14:47:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:47:08 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:47:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:47:13 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:47:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/08 14:47:19 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 14:47:24 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 14:47:29 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 14:47:34 Waiting for app: TestN1QLPosParams to get deployed
2021/12/08 14:47:34 App: TestN1QLPosParams got deployed
2021/12/08 14:47:34 Starting bucket ops to source bucket
2021/12/08 14:47:44 Waiting for dst bucket item count to get to: 5000 curr count: 520
2021/12/08 14:47:49 Waiting for dst bucket item count to get to: 5000 curr count: 1594
2021/12/08 14:47:54 Waiting for dst bucket item count to get to: 5000 curr count: 3574
2021/12/08 14:47:59 Waiting for dst bucket item count to get to: 5000 curr count: 3574
2021/12/08 14:47:59 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 14:47:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:47:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:48:04 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 14:48:09 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 14:48:14 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 14:48:19 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/08 14:48:19 App: TestN1QLPosParams got un-deployed
2021/12/08 14:48:19 No eventing-con process running
2021/12/08 14:48:19 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/08 14:48:21 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:48:31 Waiting for dst bucket item count to get to: 0 curr count: 4324
2021/12/08 14:48:36 Waiting for dst bucket item count to get to: 0 curr count: 4324
2021/12/08 14:48:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:48:47 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 14:48:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.34s)
=== RUN   TestN1QLExhaustConnPool
2021/12/08 14:48:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 14:48:52 App: TestN1QLExhaustConnPool got un-deployed
2021/12/08 14:48:52 No eventing-con process running
2021/12/08 14:48:52 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/08 14:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:49:01 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:49:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:49:06 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:49:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/08 14:49:12 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 14:49:17 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 14:49:22 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 14:49:27 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/08 14:49:27 App: TestN1QLExhaustConnPool got deployed
2021/12/08 14:49:27 Starting bucket ops to source bucket
2021/12/08 14:49:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/08 14:49:38 Waiting for dst bucket item count to get to: 100 curr count: 32
2021/12/08 14:49:43 Waiting for dst bucket item count to get to: 100 curr count: 32
2021/12/08 14:49:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/08 14:49:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:49:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:49:48 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 14:49:53 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 14:49:58 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 14:50:03 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/08 14:50:03 App: TestN1QLExhaustConnPool got un-deployed
2021/12/08 14:50:03 No eventing-con process running
2021/12/08 14:50:03 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/08 14:50:05 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:50:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 14:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:50:16 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:50:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/08 14:50:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.92s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/08 14:50:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 14:50:31 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/08 14:50:31 No eventing-con process running
2021/12/08 14:50:31 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/08 14:50:33 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:50:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:50:45 Starting bucket ops to source bucket
2021/12/08 14:51:02 Starting bucket ops to source bucket
2021/12/08 14:51:19 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 10822
2021/12/08 14:51:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/08 14:51:25 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 14:51:30 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 14:51:35 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 14:51:40 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/08 14:51:40 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/08 14:51:40 Starting bucket ops to source bucket
2021/12/08 14:51:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:51:51 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:51:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:21 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:31 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/08 14:52:36 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/08 14:52:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:52:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:52:41 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 14:52:46 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 14:52:51 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 14:52:56 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/08 14:52:56 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/08 14:52:56 No eventing-con process running
2021/12/08 14:52:56 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/08 14:52:58 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:53:08 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/08 14:53:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:53:10 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:53:20 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/08 14:53:25 Waiting for dst bucket item count to get to: 0 curr count: 15001
2021/12/08 14:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (178.20s)
=== RUN   TestN1QLQueryConsistency
2021/12/08 14:53:30 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 14:53:30 App: n1qlquery_consistency got un-deployed
2021/12/08 14:53:30 No eventing-con process running
2021/12/08 14:53:30 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/08 14:53:31 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:53:38 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:53:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:53:43 Starting bucket ops to source bucket
2021/12/08 14:53:46 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:53:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/08 14:53:52 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 14:53:57 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 14:54:02 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 14:54:07 Waiting for app: n1qlquery_consistency to get deployed
2021/12/08 14:54:07 App: n1qlquery_consistency got deployed
2021/12/08 14:54:12 Waiting for dst bucket item count to get to: 5000 curr count: 92
2021/12/08 14:54:17 Waiting for dst bucket item count to get to: 5000 curr count: 2592
2021/12/08 14:54:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/08 14:54:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/08 14:54:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:54:22 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 14:54:27 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 14:54:32 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 14:54:37 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/08 14:54:37 App: n1qlquery_consistency got un-deployed
2021/12/08 14:54:37 No eventing-con process running
2021/12/08 14:54:37 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/08 14:54:39 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:54:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 14:54:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/08 14:54:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:54:56 Context: Bucket flush, response code: 200 dump: 
2021/12/08 14:55:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/08 14:55:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.50s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1547.016s

N1ql tests took 0 hours, 25 minutes and 51 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/08 14:56:41 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:51152->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:51152->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/08 14:57:00 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 14:57:00 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/08 14:57:00 No eventing-con process running
2021/12/08 14:57:00 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/08 14:57:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:57:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 14:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 14:57:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:57:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 14:57:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:57:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 14:57:18 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 14:57:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 14:57:24 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 14:57:29 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 14:57:34 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 14:57:39 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/08 14:57:39 App: TestDuplicateWithEventingRebIn got deployed
2021/12/08 14:57:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 14:57:39 Starting bucket ops to source bucket
2021/12/08 14:57:54 Removing node(s): [] from the cluster
2021/12/08 14:57:59 Rebalance progress: 0
2021/12/08 14:58:04 Rebalance progress: 0
2021/12/08 14:58:09 Rebalance progress: 25
2021/12/08 14:58:14 Rebalance progress: 25
2021/12/08 14:58:19 Rebalance progress: 25
2021/12/08 14:58:24 Rebalance progress: 25
2021/12/08 14:58:29 Rebalance progress: 75
2021/12/08 14:58:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 14:58:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 14:58:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 14:58:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 14:58:39 Rebalance progress: 25
2021/12/08 14:58:44 Rebalance progress: 25
2021/12/08 14:58:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:58:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 14:58:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 14:58:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 14:58:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 14:58:49 Waiting for src bucket item count to get to: 200000 curr count: 100250
2021/12/08 14:58:54 Waiting for src bucket item count to get to: 200000 curr count: 117099
2021/12/08 14:58:59 Waiting for src bucket item count to get to: 200000 curr count: 117099
2021/12/08 14:59:04 Waiting for src bucket item count to get to: 200000 curr count: 133863
2021/12/08 14:59:09 Waiting for src bucket item count to get to: 200000 curr count: 133863
2021/12/08 14:59:14 Waiting for src bucket item count to get to: 200000 curr count: 150721
2021/12/08 14:59:19 Waiting for src bucket item count to get to: 200000 curr count: 150721
2021/12/08 14:59:24 Waiting for src bucket item count to get to: 200000 curr count: 167456
2021/12/08 14:59:29 Waiting for src bucket item count to get to: 200000 curr count: 167456
2021/12/08 14:59:34 Waiting for src bucket item count to get to: 200000 curr count: 184147
2021/12/08 14:59:39 Waiting for src bucket item count to get to: 200000 curr count: 184147
2021/12/08 14:59:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 14:59:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 14:59:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 14:59:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 14:59:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 14:59:59 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 15:00:04 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/08 15:00:04 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/08 15:00:04 No eventing-con process running
2021/12/08 15:00:04 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/08 15:00:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:00:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:00:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:00:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:00:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:00:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:00:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:00:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (218.02s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/08 15:00:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:00:33 Removing node(s): [] from the cluster
2021/12/08 15:00:38 Rebalance progress: 100
2021/12/08 15:00:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 15:00:43 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/08 15:00:43 No eventing-con process running
2021/12/08 15:00:43 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/08 15:00:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:00:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:00:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:00:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:00:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:00:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:01:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:01:02 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:01:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 15:01:08 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 15:01:13 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 15:01:18 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 15:01:23 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/08 15:01:23 App: TestDuplicateWithEventingRebOut got deployed
2021/12/08 15:01:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:01:23 Starting bucket ops to source bucket
2021/12/08 15:01:28 Rebalance progress: 25
2021/12/08 15:01:33 Rebalance progress: 25
2021/12/08 15:01:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:01:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 15:01:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:01:38 Waiting for src bucket item count to get to: 200000 curr count: 10091
2021/12/08 15:01:43 Waiting for src bucket item count to get to: 200000 curr count: 26922
2021/12/08 15:01:48 Waiting for src bucket item count to get to: 200000 curr count: 26922
2021/12/08 15:01:53 Waiting for src bucket item count to get to: 200000 curr count: 43679
2021/12/08 15:01:58 Waiting for src bucket item count to get to: 200000 curr count: 43679
2021/12/08 15:02:03 Waiting for src bucket item count to get to: 200000 curr count: 60691
2021/12/08 15:02:08 Waiting for src bucket item count to get to: 200000 curr count: 60691
2021/12/08 15:02:13 Waiting for src bucket item count to get to: 200000 curr count: 60691
2021/12/08 15:02:18 Waiting for src bucket item count to get to: 200000 curr count: 77163
2021/12/08 15:02:23 Waiting for src bucket item count to get to: 200000 curr count: 77163
2021/12/08 15:02:28 Waiting for src bucket item count to get to: 200000 curr count: 93769
2021/12/08 15:02:33 Waiting for src bucket item count to get to: 200000 curr count: 93769
2021/12/08 15:02:38 Waiting for src bucket item count to get to: 200000 curr count: 110660
2021/12/08 15:02:43 Waiting for src bucket item count to get to: 200000 curr count: 110660
2021/12/08 15:02:48 Waiting for src bucket item count to get to: 200000 curr count: 127578
2021/12/08 15:02:53 Waiting for src bucket item count to get to: 200000 curr count: 127578
2021/12/08 15:02:58 Waiting for src bucket item count to get to: 200000 curr count: 144647
2021/12/08 15:03:03 Waiting for src bucket item count to get to: 200000 curr count: 144647
2021/12/08 15:03:08 Waiting for src bucket item count to get to: 200000 curr count: 161443
2021/12/08 15:03:13 Waiting for src bucket item count to get to: 200000 curr count: 161443
2021/12/08 15:03:18 Waiting for src bucket item count to get to: 200000 curr count: 178065
2021/12/08 15:03:23 Waiting for src bucket item count to get to: 200000 curr count: 178065
2021/12/08 15:03:28 Waiting for src bucket item count to get to: 200000 curr count: 194748
2021/12/08 15:03:33 Waiting for src bucket item count to get to: 200000 curr count: 194748
2021/12/08 15:03:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 15:03:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 15:03:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:03:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 15:03:48 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 15:03:53 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 15:03:58 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/08 15:03:58 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/08 15:03:58 No eventing-con process running
2021/12/08 15:03:58 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/08 15:04:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:04:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:04:10 Waiting for dst bucket item count to get to: 0 curr count: 4648
2021/12/08 15:04:15 Waiting for dst bucket item count to get to: 0 curr count: 4648
2021/12/08 15:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:04:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:04:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:04:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:04:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.89s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/08 15:04:37 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 15:04:37 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/08 15:04:37 No eventing-con process running
2021/12/08 15:04:37 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/08 15:04:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:04:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:04:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:04:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:04:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:04:55 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:04:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 15:05:01 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 15:05:06 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 15:05:11 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 15:05:16 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/08 15:05:16 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/08 15:05:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:05:16 Starting bucket ops to source bucket
2021/12/08 15:05:17 Removing node(s): [] from the cluster
2021/12/08 15:05:22 Rebalance progress: 25
2021/12/08 15:05:27 Rebalance progress: 25
2021/12/08 15:05:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:05:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:05:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/08 15:05:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:05:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:05:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:05:43 Rebalance progress: 16.66666666666667
2021/12/08 15:05:48 Rebalance progress: 16.66666666666667
2021/12/08 15:05:53 Rebalance progress: 16.66666666666667
2021/12/08 15:05:58 Rebalance progress: 72.22222222222223
2021/12/08 15:06:03 Rebalance progress: 72.22222222222223
2021/12/08 15:06:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:06:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:06:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:06:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:06:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:06:14 Rebalance progress: 16.66666666666667
2021/12/08 15:06:19 Rebalance progress: 16.66666666666667
2021/12/08 15:06:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:06:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:06:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:06:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:06:24 Waiting for src bucket item count to get to: 200000 curr count: 97565
2021/12/08 15:06:29 Waiting for src bucket item count to get to: 200000 curr count: 97565
2021/12/08 15:06:34 Waiting for src bucket item count to get to: 200000 curr count: 113937
2021/12/08 15:06:39 Waiting for src bucket item count to get to: 200000 curr count: 113937
2021/12/08 15:06:44 Waiting for src bucket item count to get to: 200000 curr count: 130908
2021/12/08 15:06:49 Waiting for src bucket item count to get to: 200000 curr count: 130908
2021/12/08 15:06:54 Waiting for src bucket item count to get to: 200000 curr count: 147263
2021/12/08 15:06:59 Waiting for src bucket item count to get to: 200000 curr count: 147263
2021/12/08 15:07:04 Waiting for src bucket item count to get to: 200000 curr count: 163815
2021/12/08 15:07:09 Waiting for src bucket item count to get to: 200000 curr count: 163815
2021/12/08 15:07:14 Waiting for src bucket item count to get to: 200000 curr count: 179625
2021/12/08 15:07:19 Waiting for src bucket item count to get to: 200000 curr count: 179625
2021/12/08 15:07:24 Waiting for src bucket item count to get to: 200000 curr count: 195892
2021/12/08 15:07:29 Waiting for src bucket item count to get to: 200000 curr count: 195892
2021/12/08 15:07:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 15:07:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 15:07:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:07:39 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 15:07:44 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 15:07:49 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 15:07:54 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/08 15:07:54 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/08 15:07:54 No eventing-con process running
2021/12/08 15:07:54 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/08 15:07:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:08:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:08:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:08:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:08:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.67s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/08 15:08:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 15:08:27 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/08 15:08:27 No eventing-con process running
2021/12/08 15:08:27 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/08 15:08:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:08:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:08:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:08:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:08:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:08:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:08:46 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:08:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 15:08:52 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 15:08:57 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 15:09:02 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 15:09:07 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/08 15:09:07 App: TestDuplicateWithKVRebIn got deployed
2021/12/08 15:09:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 15:09:07 Starting bucket ops to source bucket
2021/12/08 15:09:08 Removing node(s): [] from the cluster
2021/12/08 15:09:13 Rebalance progress: 16.66666666666667
2021/12/08 15:09:18 Rebalance progress: 16.66666666666667
2021/12/08 15:09:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:09:23 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:09:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:09:28 Rebalance progress: 16.66666666666667
2021/12/08 15:09:33 Rebalance progress: 16.66666666666667
2021/12/08 15:09:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:09:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:09:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:09:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:09:38 Waiting for src bucket item count to get to: 200000 curr count: 34925
2021/12/08 15:09:43 Waiting for src bucket item count to get to: 200000 curr count: 50973
2021/12/08 15:09:48 Waiting for src bucket item count to get to: 200000 curr count: 50973
2021/12/08 15:09:53 Waiting for src bucket item count to get to: 200000 curr count: 67340
2021/12/08 15:09:58 Waiting for src bucket item count to get to: 200000 curr count: 67340
2021/12/08 15:10:03 Waiting for src bucket item count to get to: 200000 curr count: 82959
2021/12/08 15:10:08 Waiting for src bucket item count to get to: 200000 curr count: 82959
2021/12/08 15:10:13 Waiting for src bucket item count to get to: 200000 curr count: 99014
2021/12/08 15:10:18 Waiting for src bucket item count to get to: 200000 curr count: 99014
2021/12/08 15:10:23 Waiting for src bucket item count to get to: 200000 curr count: 115130
2021/12/08 15:10:28 Waiting for src bucket item count to get to: 200000 curr count: 115130
2021/12/08 15:10:33 Waiting for src bucket item count to get to: 200000 curr count: 131554
2021/12/08 15:10:38 Waiting for src bucket item count to get to: 200000 curr count: 131554
2021/12/08 15:10:43 Waiting for src bucket item count to get to: 200000 curr count: 131554
2021/12/08 15:10:48 Waiting for src bucket item count to get to: 200000 curr count: 147866
2021/12/08 15:10:53 Waiting for src bucket item count to get to: 200000 curr count: 147866
2021/12/08 15:10:58 Waiting for src bucket item count to get to: 200000 curr count: 164183
2021/12/08 15:11:03 Waiting for src bucket item count to get to: 200000 curr count: 164183
2021/12/08 15:11:08 Waiting for src bucket item count to get to: 200000 curr count: 179858
2021/12/08 15:11:13 Waiting for src bucket item count to get to: 200000 curr count: 179858
2021/12/08 15:11:18 Waiting for src bucket item count to get to: 200000 curr count: 195888
2021/12/08 15:11:23 Waiting for src bucket item count to get to: 200000 curr count: 195888
2021/12/08 15:11:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 15:11:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 15:11:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:11:33 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 15:11:38 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 15:11:43 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 15:11:48 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/08 15:11:48 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/08 15:11:48 No eventing-con process running
2021/12/08 15:11:48 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/08 15:11:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:11:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:11:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:11:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:12:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:12:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.26s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/08 15:12:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 15:12:12 Removing node(s): [] from the cluster
2021/12/08 15:12:17 Rebalance progress: 100
2021/12/08 15:12:22 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 15:12:22 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/08 15:12:22 No eventing-con process running
2021/12/08 15:12:22 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/08 15:12:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:12:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:12:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:12:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:12:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:12:40 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:12:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 15:12:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 15:12:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 15:12:56 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 15:13:01 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/08 15:13:01 App: TestDuplicateWithKVRebOut got deployed
2021/12/08 15:13:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:13:01 Starting bucket ops to source bucket
2021/12/08 15:13:07 Rebalance progress: 16.66666666666667
2021/12/08 15:13:12 Rebalance progress: 16.66666666666667
2021/12/08 15:13:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:13:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:13:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:13:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:13:17 Waiting for src bucket item count to get to: 200000 curr count: 12081
2021/12/08 15:13:22 Waiting for src bucket item count to get to: 200000 curr count: 28711
2021/12/08 15:13:27 Waiting for src bucket item count to get to: 200000 curr count: 28711
2021/12/08 15:13:32 Waiting for src bucket item count to get to: 200000 curr count: 45134
2021/12/08 15:13:37 Waiting for src bucket item count to get to: 200000 curr count: 45134
2021/12/08 15:13:42 Waiting for src bucket item count to get to: 200000 curr count: 61358
2021/12/08 15:13:47 Waiting for src bucket item count to get to: 200000 curr count: 61358
2021/12/08 15:13:52 Waiting for src bucket item count to get to: 200000 curr count: 61358
2021/12/08 15:13:57 Waiting for src bucket item count to get to: 200000 curr count: 77183
2021/12/08 15:14:02 Waiting for src bucket item count to get to: 200000 curr count: 77183
2021/12/08 15:14:07 Waiting for src bucket item count to get to: 200000 curr count: 93655
2021/12/08 15:14:12 Waiting for src bucket item count to get to: 200000 curr count: 93655
2021/12/08 15:14:17 Waiting for src bucket item count to get to: 200000 curr count: 110335
2021/12/08 15:14:22 Waiting for src bucket item count to get to: 200000 curr count: 110335
2021/12/08 15:14:27 Waiting for src bucket item count to get to: 200000 curr count: 127048
2021/12/08 15:14:32 Waiting for src bucket item count to get to: 200000 curr count: 127048
2021/12/08 15:14:37 Waiting for src bucket item count to get to: 200000 curr count: 143413
2021/12/08 15:14:42 Waiting for src bucket item count to get to: 200000 curr count: 143413
2021/12/08 15:14:47 Waiting for src bucket item count to get to: 200000 curr count: 160019
2021/12/08 15:14:52 Waiting for src bucket item count to get to: 200000 curr count: 160019
2021/12/08 15:14:57 Waiting for src bucket item count to get to: 200000 curr count: 176268
2021/12/08 15:15:02 Waiting for src bucket item count to get to: 200000 curr count: 176268
2021/12/08 15:15:07 Waiting for src bucket item count to get to: 200000 curr count: 192571
2021/12/08 15:15:12 Waiting for src bucket item count to get to: 200000 curr count: 192571
2021/12/08 15:15:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 15:15:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 15:15:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:15:22 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 15:15:27 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 15:15:32 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 15:15:37 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/08 15:15:37 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/08 15:15:37 No eventing-con process running
2021/12/08 15:15:37 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/08 15:15:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:15:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:15:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:15:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:15:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:16:00 Waiting for dst bucket item count to get to: 0 curr count: 117996
2021/12/08 15:16:05 Waiting for dst bucket item count to get to: 0 curr count: 117996
2021/12/08 15:16:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (238.92s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/08 15:16:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 15:16:16 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/08 15:16:16 No eventing-con process running
2021/12/08 15:16:16 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/08 15:16:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:16:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:16:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:16:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:16:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:16:34 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:16:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 15:16:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 15:16:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 15:16:50 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 15:16:55 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/08 15:16:55 App: TestDuplicateWithKVSwapReb got deployed
2021/12/08 15:16:55 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 15:16:55 Starting bucket ops to source bucket
2021/12/08 15:16:55 Removing node(s): [] from the cluster
2021/12/08 15:17:00 Rebalance progress: 16.66666666666667
2021/12/08 15:17:05 Rebalance progress: 16.66666666666667
2021/12/08 15:17:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:17:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:17:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 15:17:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:17:16 Rebalance progress: 16.66666666666667
2021/12/08 15:17:21 Rebalance progress: 16.66666666666667
2021/12/08 15:17:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:17:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:17:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:17:31 Rebalance progress: 16.66666666666667
2021/12/08 15:17:36 Rebalance progress: 16.66666666666667
2021/12/08 15:17:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:17:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:17:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:17:41 Waiting for src bucket item count to get to: 200000 curr count: 69600
2021/12/08 15:17:46 Waiting for src bucket item count to get to: 200000 curr count: 69600
2021/12/08 15:17:51 Waiting for src bucket item count to get to: 200000 curr count: 85300
2021/12/08 15:17:56 Waiting for src bucket item count to get to: 200000 curr count: 85300
2021/12/08 15:18:01 Waiting for src bucket item count to get to: 200000 curr count: 101702
2021/12/08 15:18:06 Waiting for src bucket item count to get to: 200000 curr count: 101702
2021/12/08 15:18:11 Waiting for src bucket item count to get to: 200000 curr count: 118356
2021/12/08 15:18:16 Waiting for src bucket item count to get to: 200000 curr count: 118356
2021/12/08 15:18:21 Waiting for src bucket item count to get to: 200000 curr count: 135058
2021/12/08 15:18:26 Waiting for src bucket item count to get to: 200000 curr count: 135058
2021/12/08 15:18:31 Waiting for src bucket item count to get to: 200000 curr count: 151693
2021/12/08 15:18:36 Waiting for src bucket item count to get to: 200000 curr count: 151693
2021/12/08 15:18:41 Waiting for src bucket item count to get to: 200000 curr count: 151693
2021/12/08 15:18:46 Waiting for src bucket item count to get to: 200000 curr count: 168363
2021/12/08 15:18:51 Waiting for src bucket item count to get to: 200000 curr count: 168363
2021/12/08 15:18:56 Waiting for src bucket item count to get to: 200000 curr count: 184205
2021/12/08 15:19:01 Waiting for src bucket item count to get to: 200000 curr count: 184205
2021/12/08 15:19:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 15:19:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 15:19:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:19:11 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 15:19:16 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 15:19:21 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 15:19:26 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/08 15:19:26 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/08 15:19:26 No eventing-con process running
2021/12/08 15:19:26 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/08 15:19:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:19:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/08 15:19:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:19:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:19:50 Waiting for dst bucket item count to get to: 0 curr count: 7506
2021/12/08 15:19:55 Waiting for dst bucket item count to get to: 0 curr count: 7506
2021/12/08 15:19:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.49s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/08 15:20:05 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:20:05 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:20:05 No eventing-con process running
2021/12/08 15:20:05 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 15:20:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:20:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:20:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:20:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:20:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:20:23 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:20:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 15:20:30 Starting bucket ops to source bucket
2021/12/08 15:20:35 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:20:40 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:20:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:20:45 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:20:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:20:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:20:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:20:50 Rebalance progress: 16.66666666666667
2021/12/08 15:20:55 Rebalance progress: 16.66666666666667
2021/12/08 15:21:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:21:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:21:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:21:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:21:05 Rebalance progress: 16.66666666666667
2021/12/08 15:21:10 Rebalance progress: 16.66666666666667
2021/12/08 15:21:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:21:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/08 15:21:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 15:21:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:21:26 Rebalance progress: 12.5
2021/12/08 15:21:31 Rebalance progress: 12.5
2021/12/08 15:21:36 Rebalance progress: 12.5
2021/12/08 15:21:41 Rebalance progress: 70.83333333333334
2021/12/08 15:21:46 Rebalance progress: 70.83333333333334
2021/12/08 15:21:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:21:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:21:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:21:56 Rebalance progress: 12.5
2021/12/08 15:22:01 Rebalance progress: 12.5
2021/12/08 15:22:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:22:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:22:11 Rebalance progress: 12.5
2021/12/08 15:22:16 Rebalance progress: 12.5
2021/12/08 15:22:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:22:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 15:22:26 Rebalance progress: 12.5
2021/12/08 15:22:31 Rebalance progress: 12.5
2021/12/08 15:22:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:22:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:22:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:22:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:22:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:22:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:22:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:22:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:22:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 15:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:23:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:23:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12627
2021/12/08 15:23:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:23:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:23:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:23:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.77s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/08 15:23:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 15:23:25 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/08 15:23:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:23:25 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/08 15:23:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:23:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:23:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:23:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:23:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:23:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:23:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:23:43 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:23:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/08 15:23:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:23:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 15:23:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 15:24:04 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/08 15:24:04 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:24:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:24:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:24:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 15:24:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:24:11 Rebalance progress: 12.5
2021/12/08 15:24:16 Rebalance progress: 12.5
2021/12/08 15:24:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:24:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 15:24:26 Rebalance progress: 12.5
2021/12/08 15:24:31 Rebalance progress: 12.5
2021/12/08 15:24:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:24:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:24:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:24:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 15:24:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 15:24:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 15:24:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/08 15:24:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/08 15:24:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:24:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/08 15:24:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:25:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:25:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4615
2021/12/08 15:25:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:25:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:25:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:25:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.66s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/08 15:25:24 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 15:25:24 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 15:25:24 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:25:24 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/08 15:25:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:25:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:25:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:25:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:25:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:25:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:25:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:25:43 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:25:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/08 15:25:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:25:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 15:25:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 15:26:04 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/08 15:26:04 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:26:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:26:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:26:10 Rebalance progress: 12.5
2021/12/08 15:26:15 Rebalance progress: 12.5
2021/12/08 15:26:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:26:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:26:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:26:25 Rebalance progress: 12.5
2021/12/08 15:26:30 Rebalance progress: 12.5
2021/12/08 15:26:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:26:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 15:26:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:26:40 Rebalance progress: 12.5
2021/12/08 15:26:45 Rebalance progress: 12.5
2021/12/08 15:26:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:26:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:26:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:26:56 Rebalance progress: 12.5
2021/12/08 15:27:01 Rebalance progress: 12.5
2021/12/08 15:27:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:27:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:27:11 Rebalance progress: 12.5
2021/12/08 15:27:16 Rebalance progress: 12.5
2021/12/08 15:27:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:27:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 15:27:26 Rebalance progress: 12.5
2021/12/08 15:27:31 Rebalance progress: 12.5
2021/12/08 15:27:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:27:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:27:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:27:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 15:27:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 15:27:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 15:27:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/08 15:27:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/08 15:27:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:27:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/08 15:27:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:28:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10656
2021/12/08 15:28:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:28:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:28:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (180.07s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/08 15:28:24 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 15:28:25 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/08 15:28:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:28:25 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/08 15:28:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:28:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:28:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:28:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:28:38 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:28:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 15:28:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:28:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 15:28:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 15:28:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/08 15:28:59 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:28:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:28:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:29:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:29:05 Rebalance progress: 12.5
2021/12/08 15:29:10 Rebalance progress: 12.5
2021/12/08 15:29:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:29:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:29:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:29:20 Rebalance progress: 12.5
2021/12/08 15:29:25 Rebalance progress: 12.5
2021/12/08 15:29:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:29:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:29:35 Rebalance progress: 12.5
2021/12/08 15:29:40 Rebalance progress: 12.5
2021/12/08 15:29:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:29:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:29:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:29:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 15:29:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 15:30:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 15:30:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/08 15:30:05 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/08 15:30:05 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:30:05 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/08 15:30:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:30:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:30:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/08 15:30:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:30:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:30:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:30:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/08 15:30:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (134.26s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/08 15:30:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:30:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:30:39 Rebalance progress: 100
2021/12/08 15:30:44 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 15:30:44 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/08 15:30:44 No eventing-con process running
2021/12/08 15:30:44 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/08 15:30:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:30:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:30:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:30:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:30:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:30:58 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:30:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 15:31:04 Starting bucket ops to source bucket
2021/12/08 15:31:09 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 15:31:14 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 15:31:19 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/08 15:31:19 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:31:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:31:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:31:24 Rebalance progress: 12.5
2021/12/08 15:31:29 Rebalance progress: 12.5
2021/12/08 15:31:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:31:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/08 15:31:34 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 15:31:39 Rebalance progress: 12.5
2021/12/08 15:31:44 Rebalance progress: 12.5
2021/12/08 15:31:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:31:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:31:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:31:54 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 15:32:00 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 15:32:05 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 15:32:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/08 15:32:10 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/08 15:32:10 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:32:10 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/08 15:32:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:32:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:32:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/08 15:32:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/08 15:32:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:32:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:32:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:32:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/08 15:32:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.31s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/08 15:32:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:32:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:32:48 Rebalance progress: 100
2021/12/08 15:32:53 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 15:32:53 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/08 15:32:53 No eventing-con process running
2021/12/08 15:32:53 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/08 15:32:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:33:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:33:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:33:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:33:07 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:33:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/08 15:33:13 Starting bucket ops to source bucket
2021/12/08 15:33:18 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 15:33:23 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 15:33:28 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/08 15:33:28 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:33:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:33:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:33:34 Rebalance progress: 12.5
2021/12/08 15:33:39 Rebalance progress: 12.5
2021/12/08 15:33:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:33:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/08 15:33:44 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 15:33:49 Rebalance progress: 12.5
2021/12/08 15:33:54 Rebalance progress: 12.5
2021/12/08 15:33:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:33:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:33:59 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/08 15:34:04 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 15:34:09 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 15:34:14 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 15:34:19 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/08 15:34:19 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/08 15:34:19 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:34:19 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/08 15:34:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:34:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:34:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38534
2021/12/08 15:34:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38534
2021/12/08 15:34:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:34:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:34:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:34:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 38534
2021/12/08 15:34:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.88s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/08 15:34:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 15:34:58 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/08 15:34:58 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:34:58 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/08 15:35:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:35:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:35:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:35:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:35:17 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:35:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:35:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:35:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 15:35:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:35:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 15:35:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 15:35:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/08 15:35:39 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/08 15:35:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 15:35:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 15:35:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/08 15:35:54 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:35:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:35:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:35:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:36:00 Rebalance progress: 12.5
2021/12/08 15:36:05 Rebalance progress: 12.5
2021/12/08 15:36:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:36:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:36:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:36:15 Rebalance progress: 12.5
2021/12/08 15:36:20 Rebalance progress: 12.5
2021/12/08 15:36:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:36:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 15:36:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:36:30 Rebalance progress: 12.5
2021/12/08 15:36:35 Rebalance progress: 12.5
2021/12/08 15:36:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:36:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:36:46 Rebalance progress: 12.5
2021/12/08 15:36:51 Rebalance progress: 12.5
2021/12/08 15:36:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:36:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:36:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:37:01 Rebalance progress: 12.5
2021/12/08 15:37:06 Rebalance progress: 12.5
2021/12/08 15:37:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:37:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 15:37:16 Rebalance progress: 12.5
2021/12/08 15:37:21 Rebalance progress: 12.5
2021/12/08 15:37:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:37:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:37:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:37:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 15:37:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 15:37:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 15:37:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/08 15:37:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/08 15:37:46 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:37:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/08 15:37:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:37:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:37:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:37:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:38:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12192
2021/12/08 15:38:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:38:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:38:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 15:38:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 15:38:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 15:38:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/08 15:38:26 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/08 15:38:26 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:38:27 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/08 15:38:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:38:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:38:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:38:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:38:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.09s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/08 15:38:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 15:38:50 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/08 15:38:50 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:38:50 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/08 15:38:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:38:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:38:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:38:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:39:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:39:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:39:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:39:09 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:39:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:39:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:39:10 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:39:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 15:39:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:39:21 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 15:39:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 15:39:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/08 15:39:31 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/08 15:39:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 15:39:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 15:39:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/08 15:39:46 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:39:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:39:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:39:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:39:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:39:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/08 15:39:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:39:52 Rebalance progress: 12.5
2021/12/08 15:39:57 Rebalance progress: 12.5
2021/12/08 15:40:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:40:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:40:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:40:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 15:40:07 Rebalance progress: 12.5
2021/12/08 15:40:12 Rebalance progress: 12.5
2021/12/08 15:40:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:40:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:40:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:40:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:40:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 15:40:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 15:40:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 15:40:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/08 15:40:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/08 15:40:37 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:40:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/08 15:40:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:40:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:40:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1084
2021/12/08 15:40:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1084
2021/12/08 15:40:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:40:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:41:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:41:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6132
2021/12/08 15:41:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:41:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:41:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 15:41:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 15:41:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 15:41:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/08 15:41:28 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/08 15:41:28 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:41:28 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/08 15:41:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:41:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:41:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:41:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:41:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:41:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.63s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/08 15:41:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 15:41:52 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/08 15:41:52 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:41:52 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/08 15:41:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:41:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:41:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:42:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:42:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:42:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:42:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 15:42:10 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/08 15:42:10 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:42:10 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/08 15:42:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:42:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:42:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:42:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:42:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:42:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:42:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:42:24 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:42:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:42:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:42:25 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:42:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/08 15:42:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:42:36 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 15:42:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 15:42:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/08 15:42:46 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/08 15:42:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 15:42:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 15:43:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/08 15:43:01 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:43:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:43:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:43:07 Rebalance progress: 12.5
2021/12/08 15:43:12 Rebalance progress: 12.5
2021/12/08 15:43:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:43:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:43:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:43:23 Rebalance progress: 12.5
2021/12/08 15:43:28 Rebalance progress: 12.5
2021/12/08 15:43:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:43:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:43:38 Rebalance progress: 12.5
2021/12/08 15:43:43 Rebalance progress: 12.5
2021/12/08 15:43:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:43:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:43:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:43:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 15:43:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 15:44:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 15:44:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/08 15:44:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/08 15:44:08 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:44:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/08 15:44:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:44:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:44:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3373
2021/12/08 15:44:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3373
2021/12/08 15:44:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:44:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:44:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:44:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7666
2021/12/08 15:44:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:44:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:44:44 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 15:44:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 15:44:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 15:44:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/08 15:44:59 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/08 15:44:59 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:44:59 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/08 15:45:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:45:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:45:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:45:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:45:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:45:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.51s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/08 15:45:22 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:45:22 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:45:22 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:45:22 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 15:45:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:45:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:45:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:45:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:45:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:45:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:45:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:45:41 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:45:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 15:45:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:45:52 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:45:57 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:46:02 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:46:02 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:46:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:46:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:46:22 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:46:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:46:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:46:58 Rebalance progress: 12.5
2021/12/08 15:47:03 Rebalance progress: 12.5
2021/12/08 15:47:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:47:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:47:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:47:14 Rebalance progress: 12.5
2021/12/08 15:47:19 Rebalance progress: 12.5
2021/12/08 15:47:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:47:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:47:29 Rebalance progress: 12.5
2021/12/08 15:47:34 Rebalance progress: 12.5
2021/12/08 15:47:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:47:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:47:44 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:47:49 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:47:54 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:47:59 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:47:59 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:47:59 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:47:59 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 15:48:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:48:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:48:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11212
2021/12/08 15:48:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11212
2021/12/08 15:48:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:48:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:48:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:48:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11212
2021/12/08 15:48:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.57s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/08 15:48:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:48:38 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:48:38 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:48:38 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/08 15:48:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:48:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:48:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:48:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:48:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:48:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:48:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:48:56 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:48:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/08 15:49:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:49:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:49:12 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:49:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/08 15:49:17 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:49:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:49:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:49:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:49:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:49:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:49:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:50:09 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:50:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:50:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:50:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:50:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:50:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:51:10 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:51:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:51:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:51:41 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:51:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:51:57 Rebalance progress: 12.5
2021/12/08 15:52:02 Rebalance progress: 12.5
2021/12/08 15:52:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:52:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:52:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:52:12 Rebalance progress: 12.5
2021/12/08 15:52:17 Rebalance progress: 12.5
2021/12/08 15:52:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:52:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:52:27 Rebalance progress: 12.5
2021/12/08 15:52:32 Rebalance progress: 12.5
2021/12/08 15:52:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:52:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:52:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:52:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:52:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:52:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:52:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/08 15:52:58 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/08 15:52:58 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:52:58 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/08 15:52:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:53:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:53:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21496
2021/12/08 15:53:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:53:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:53:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:53:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21496
2021/12/08 15:53:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21496
2021/12/08 15:53:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.36s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/08 15:53:36 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 15:53:36 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/08 15:53:36 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:53:36 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/08 15:53:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:53:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:53:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:53:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:53:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:53:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:53:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:53:55 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:53:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 15:54:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:54:06 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 15:54:11 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 15:54:16 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/08 15:54:16 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:54:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:54:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:54:21 Rebalance progress: 12.5
2021/12/08 15:54:26 Rebalance progress: 12.5
2021/12/08 15:54:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:54:31 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:54:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:54:46 Rebalance progress: 12.5
2021/12/08 15:54:51 Rebalance progress: 12.5
2021/12/08 15:54:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:54:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:54:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 15:54:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:55:02 Rebalance progress: 12.5
2021/12/08 15:55:07 Rebalance progress: 12.5
2021/12/08 15:55:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:55:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:55:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:55:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:55:17 Rebalance progress: 12.5
2021/12/08 15:55:22 Rebalance progress: 12.5
2021/12/08 15:55:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:55:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:55:33 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 15:55:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 15:55:43 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 15:55:48 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/08 15:55:48 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/08 15:55:48 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:55:48 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/08 15:55:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:55:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:55:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:55:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:56:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8662
2021/12/08 15:56:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.99s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/08 15:56:16 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 15:56:16 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/08 15:56:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:56:16 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/08 15:56:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:56:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:56:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:56:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:56:35 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:56:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/08 15:56:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:56:46 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 15:56:51 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 15:56:56 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/08 15:56:56 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:56:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:56:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/08 15:56:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:57:01 Rebalance progress: 12.5
2021/12/08 15:57:06 Rebalance progress: 12.5
2021/12/08 15:57:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:57:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/08 15:57:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:57:16 Rebalance progress: 12.5
2021/12/08 15:57:21 Rebalance progress: 12.5
2021/12/08 15:57:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:57:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 15:57:32 Rebalance progress: 12.5
2021/12/08 15:57:37 Rebalance progress: 12.5
2021/12/08 15:57:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:57:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 15:57:47 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 15:57:52 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 15:57:57 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 15:58:02 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/08 15:58:02 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/08 15:58:02 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:58:02 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/08 15:58:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:58:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:58:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6065
2021/12/08 15:58:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6065
2021/12/08 15:58:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:58:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:58:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:58:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2483
2021/12/08 15:58:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2483
2021/12/08 15:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.22s)
=== RUN   TestEventingRebInWithOps
2021/12/08 15:58:45 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 15:58:45 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/08 15:58:45 eventingSetup.go:46: No eventing-con process running
2021/12/08 15:58:45 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/08 15:58:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:58:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:58:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 15:58:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:58:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:58:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:59:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 15:59:04 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 15:59:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 15:59:10 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 15:59:15 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 15:59:20 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 15:59:25 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/08 15:59:25 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/08 15:59:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 15:59:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 15:59:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 15:59:30 Rebalance progress: 12.5
2021/12/08 15:59:35 Rebalance progress: 12.5
2021/12/08 15:59:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:59:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:59:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:59:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 15:59:46 Rebalance progress: 12.5
2021/12/08 15:59:51 Rebalance progress: 12.5
2021/12/08 15:59:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 15:59:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 15:59:56 Waiting for src bucket item count to get to: 200000 curr count: 37697
2021/12/08 16:00:01 Waiting for src bucket item count to get to: 200000 curr count: 53498
2021/12/08 16:00:06 Waiting for src bucket item count to get to: 200000 curr count: 53498
2021/12/08 16:00:11 Waiting for src bucket item count to get to: 200000 curr count: 69847
2021/12/08 16:00:16 Waiting for src bucket item count to get to: 200000 curr count: 69847
2021/12/08 16:00:21 Waiting for src bucket item count to get to: 200000 curr count: 85415
2021/12/08 16:00:26 Waiting for src bucket item count to get to: 200000 curr count: 85415
2021/12/08 16:00:31 Waiting for src bucket item count to get to: 200000 curr count: 101535
2021/12/08 16:00:36 Waiting for src bucket item count to get to: 200000 curr count: 101535
2021/12/08 16:00:41 Waiting for src bucket item count to get to: 200000 curr count: 101535
2021/12/08 16:00:46 Waiting for src bucket item count to get to: 200000 curr count: 118059
2021/12/08 16:00:51 Waiting for src bucket item count to get to: 200000 curr count: 118059
2021/12/08 16:00:56 Waiting for src bucket item count to get to: 200000 curr count: 134733
2021/12/08 16:01:01 Waiting for src bucket item count to get to: 200000 curr count: 134733
2021/12/08 16:01:06 Waiting for src bucket item count to get to: 200000 curr count: 150932
2021/12/08 16:01:11 Waiting for src bucket item count to get to: 200000 curr count: 150932
2021/12/08 16:01:16 Waiting for src bucket item count to get to: 200000 curr count: 167419
2021/12/08 16:01:21 Waiting for src bucket item count to get to: 200000 curr count: 167419
2021/12/08 16:01:26 Waiting for src bucket item count to get to: 200000 curr count: 182893
2021/12/08 16:01:31 Waiting for src bucket item count to get to: 200000 curr count: 182893
2021/12/08 16:01:36 Waiting for src bucket item count to get to: 200000 curr count: 199218
2021/12/08 16:01:41 Waiting for src bucket item count to get to: 200000 curr count: 199218
2021/12/08 16:01:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:01:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:01:46 Starting bucket ops to source bucket
2021/12/08 16:05:31 Waiting for dst bucket item count to get to: 0 curr count: 12549
2021/12/08 16:05:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:05:31 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:05:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:05:36 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 16:05:41 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 16:05:46 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 16:05:51 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/08 16:05:51 App: TestEventingRebInWithOps got un-deployed
2021/12/08 16:05:51 No eventing-con process running
2021/12/08 16:05:51 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/08 16:05:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:05:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:06:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:06:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:06:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (449.45s)
=== RUN   TestEventingRebOutWithOps
2021/12/08 16:06:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 16:06:10 Removing node(s): [] from the cluster
2021/12/08 16:06:15 Rebalance progress: 100
2021/12/08 16:06:20 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 16:06:20 App: TestEventingRebOutWithOps got un-deployed
2021/12/08 16:06:20 No eventing-con process running
2021/12/08 16:06:20 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/08 16:06:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:06:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:06:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:06:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:06:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:06:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:06:39 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:06:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 16:06:45 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 16:06:50 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 16:06:55 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 16:07:00 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/08 16:07:00 App: TestEventingRebOutWithOps got deployed
2021/12/08 16:07:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:07:00 Starting bucket ops to source bucket
2021/12/08 16:07:05 Rebalance progress: 12.5
2021/12/08 16:07:10 Rebalance progress: 12.5
2021/12/08 16:07:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:07:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:07:15 Waiting for src bucket item count to get to: 200000 curr count: 14728
2021/12/08 16:07:20 Waiting for src bucket item count to get to: 200000 curr count: 30920
2021/12/08 16:07:25 Waiting for src bucket item count to get to: 200000 curr count: 30920
2021/12/08 16:07:30 Waiting for src bucket item count to get to: 200000 curr count: 47381
2021/12/08 16:07:35 Waiting for src bucket item count to get to: 200000 curr count: 47381
2021/12/08 16:07:40 Waiting for src bucket item count to get to: 200000 curr count: 63823
2021/12/08 16:07:45 Waiting for src bucket item count to get to: 200000 curr count: 63823
2021/12/08 16:07:50 Waiting for src bucket item count to get to: 200000 curr count: 63823
2021/12/08 16:07:55 Waiting for src bucket item count to get to: 200000 curr count: 79455
2021/12/08 16:08:00 Waiting for src bucket item count to get to: 200000 curr count: 79455
2021/12/08 16:08:05 Waiting for src bucket item count to get to: 200000 curr count: 95629
2021/12/08 16:08:10 Waiting for src bucket item count to get to: 200000 curr count: 95629
2021/12/08 16:08:15 Waiting for src bucket item count to get to: 200000 curr count: 112092
2021/12/08 16:08:20 Waiting for src bucket item count to get to: 200000 curr count: 112092
2021/12/08 16:08:25 Waiting for src bucket item count to get to: 200000 curr count: 128577
2021/12/08 16:08:30 Waiting for src bucket item count to get to: 200000 curr count: 128577
2021/12/08 16:08:35 Waiting for src bucket item count to get to: 200000 curr count: 145245
2021/12/08 16:08:40 Waiting for src bucket item count to get to: 200000 curr count: 145245
2021/12/08 16:08:45 Waiting for src bucket item count to get to: 200000 curr count: 161641
2021/12/08 16:08:50 Waiting for src bucket item count to get to: 200000 curr count: 161641
2021/12/08 16:08:55 Waiting for src bucket item count to get to: 200000 curr count: 176923
2021/12/08 16:09:00 Waiting for src bucket item count to get to: 200000 curr count: 176923
2021/12/08 16:09:05 Waiting for src bucket item count to get to: 200000 curr count: 193227
2021/12/08 16:09:10 Waiting for src bucket item count to get to: 200000 curr count: 193227
2021/12/08 16:09:15 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:09:15 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:09:15 Starting bucket ops to source bucket
2021/12/08 16:13:00 Waiting for dst bucket item count to get to: 0 curr count: 10423
2021/12/08 16:13:05 Waiting for dst bucket item count to get to: 0 curr count: 10423
2021/12/08 16:13:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:13:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:13:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:13:10 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 16:13:15 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 16:13:20 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 16:13:25 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/08 16:13:25 App: TestEventingRebOutWithOps got un-deployed
2021/12/08 16:13:25 No eventing-con process running
2021/12/08 16:13:25 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/08 16:13:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:13:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:13:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:13:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:13:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:13:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (453.91s)
=== RUN   TestEventingSwapRebWithOps
2021/12/08 16:13:49 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/08 16:13:49 App: TestEventingSwapRebWithOps got un-deployed
2021/12/08 16:13:49 No eventing-con process running
2021/12/08 16:13:49 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/08 16:13:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:13:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:13:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:14:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:14:07 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:14:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 16:14:13 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/08 16:14:18 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/08 16:14:23 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/08 16:14:28 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/08 16:14:28 App: TestEventingSwapRebWithOps got deployed
2021/12/08 16:14:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 16:14:28 Starting bucket ops to source bucket
2021/12/08 16:14:29 Removing node(s): [] from the cluster
2021/12/08 16:14:34 Rebalance progress: 12.5
2021/12/08 16:14:39 Rebalance progress: 12.5
2021/12/08 16:14:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:14:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 16:14:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:14:49 Rebalance progress: 12.5
2021/12/08 16:14:54 Rebalance progress: 12.5
2021/12/08 16:14:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:14:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:14:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 16:15:05 Rebalance progress: 12.5
2021/12/08 16:15:10 Rebalance progress: 12.5
2021/12/08 16:15:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:15:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:15:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:15:15 Waiting for src bucket item count to get to: 200000 curr count: 63053
2021/12/08 16:15:20 Waiting for src bucket item count to get to: 200000 curr count: 63053
2021/12/08 16:15:25 Waiting for src bucket item count to get to: 200000 curr count: 78389
2021/12/08 16:15:30 Waiting for src bucket item count to get to: 200000 curr count: 78389
2021/12/08 16:15:35 Waiting for src bucket item count to get to: 200000 curr count: 94495
2021/12/08 16:15:40 Waiting for src bucket item count to get to: 200000 curr count: 94495
2021/12/08 16:15:45 Waiting for src bucket item count to get to: 200000 curr count: 110817
2021/12/08 16:15:50 Waiting for src bucket item count to get to: 200000 curr count: 110817
2021/12/08 16:15:55 Waiting for src bucket item count to get to: 200000 curr count: 127092
2021/12/08 16:16:00 Waiting for src bucket item count to get to: 200000 curr count: 127092
2021/12/08 16:16:05 Waiting for src bucket item count to get to: 200000 curr count: 143088
2021/12/08 16:16:10 Waiting for src bucket item count to get to: 200000 curr count: 143088
2021/12/08 16:16:15 Waiting for src bucket item count to get to: 200000 curr count: 159308
2021/12/08 16:16:20 Waiting for src bucket item count to get to: 200000 curr count: 159308
2021/12/08 16:16:25 Waiting for src bucket item count to get to: 200000 curr count: 174438
2021/12/08 16:16:30 Waiting for src bucket item count to get to: 200000 curr count: 174438
2021/12/08 16:16:35 Waiting for src bucket item count to get to: 200000 curr count: 190393
2021/12/08 16:16:40 Waiting for src bucket item count to get to: 200000 curr count: 190393
2021/12/08 16:16:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:16:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:16:45 Starting bucket ops to source bucket
2021/12/08 16:20:30 Waiting for dst bucket item count to get to: 0 curr count: 9879
2021/12/08 16:20:35 Waiting for dst bucket item count to get to: 0 curr count: 9879
2021/12/08 16:20:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:20:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:20:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:20:40 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/08 16:20:45 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/08 16:20:50 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/08 16:20:55 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/08 16:20:55 App: TestEventingSwapRebWithOps got un-deployed
2021/12/08 16:20:55 No eventing-con process running
2021/12/08 16:20:55 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/08 16:20:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:21:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:21:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:21:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:21:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:21:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (449.52s)
=== RUN   TestKVRebInWithOps
2021/12/08 16:21:18 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/08 16:21:18 App: TestKVRebInWithOps got un-deployed
2021/12/08 16:21:18 No eventing-con process running
2021/12/08 16:21:18 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/08 16:21:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:21:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:21:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:21:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:21:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:21:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:21:37 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:21:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 16:21:43 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/08 16:21:48 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/08 16:21:53 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/08 16:21:58 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/08 16:21:58 App: TestKVRebInWithOps got deployed
2021/12/08 16:21:58 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 16:21:58 Starting bucket ops to source bucket
2021/12/08 16:21:58 Removing node(s): [] from the cluster
2021/12/08 16:22:03 Rebalance progress: 12.5
2021/12/08 16:22:08 Rebalance progress: 12.5
2021/12/08 16:22:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:22:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:22:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:22:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:22:18 Rebalance progress: 12.5
2021/12/08 16:22:23 Rebalance progress: 12.5
2021/12/08 16:22:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:22:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:22:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:22:28 Waiting for src bucket item count to get to: 200000 curr count: 33121
2021/12/08 16:22:33 Waiting for src bucket item count to get to: 200000 curr count: 49268
2021/12/08 16:22:38 Waiting for src bucket item count to get to: 200000 curr count: 49268
2021/12/08 16:22:43 Waiting for src bucket item count to get to: 200000 curr count: 65715
2021/12/08 16:22:48 Waiting for src bucket item count to get to: 200000 curr count: 65715
2021/12/08 16:22:53 Waiting for src bucket item count to get to: 200000 curr count: 81305
2021/12/08 16:22:59 Waiting for src bucket item count to get to: 200000 curr count: 81305
2021/12/08 16:23:04 Waiting for src bucket item count to get to: 200000 curr count: 97630
2021/12/08 16:23:09 Waiting for src bucket item count to get to: 200000 curr count: 97630
2021/12/08 16:23:14 Waiting for src bucket item count to get to: 200000 curr count: 114240
2021/12/08 16:23:19 Waiting for src bucket item count to get to: 200000 curr count: 114240
2021/12/08 16:23:24 Waiting for src bucket item count to get to: 200000 curr count: 114240
2021/12/08 16:23:29 Waiting for src bucket item count to get to: 200000 curr count: 131014
2021/12/08 16:23:34 Waiting for src bucket item count to get to: 200000 curr count: 131014
2021/12/08 16:23:39 Waiting for src bucket item count to get to: 200000 curr count: 147615
2021/12/08 16:23:44 Waiting for src bucket item count to get to: 200000 curr count: 147615
2021/12/08 16:23:49 Waiting for src bucket item count to get to: 200000 curr count: 164109
2021/12/08 16:23:54 Waiting for src bucket item count to get to: 200000 curr count: 164109
2021/12/08 16:23:59 Waiting for src bucket item count to get to: 200000 curr count: 179503
2021/12/08 16:24:04 Waiting for src bucket item count to get to: 200000 curr count: 179503
2021/12/08 16:24:09 Waiting for src bucket item count to get to: 200000 curr count: 195535
2021/12/08 16:24:14 Waiting for src bucket item count to get to: 200000 curr count: 195535
2021/12/08 16:24:19 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:24:19 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:24:19 Starting bucket ops to source bucket
2021/12/08 16:28:04 Waiting for dst bucket item count to get to: 0 curr count: 17518
2021/12/08 16:28:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:28:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:28:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:28:09 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/08 16:28:14 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/08 16:28:19 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/08 16:28:24 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/08 16:28:24 App: TestKVRebInWithOps got un-deployed
2021/12/08 16:28:24 No eventing-con process running
2021/12/08 16:28:24 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/08 16:28:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:28:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:28:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:28:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:28:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:28:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.32s)
=== RUN   TestKVRebOutWithOps
2021/12/08 16:28:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 16:28:43 Removing node(s): [] from the cluster
2021/12/08 16:28:48 Rebalance progress: 100
2021/12/08 16:28:53 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/08 16:28:53 App: TestKVRebOutWithOps got un-deployed
2021/12/08 16:28:53 No eventing-con process running
2021/12/08 16:28:53 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/08 16:28:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:29:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:29:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:29:12 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:29:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 16:29:18 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/08 16:29:23 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/08 16:29:28 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/08 16:29:33 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/08 16:29:33 App: TestKVRebOutWithOps got deployed
2021/12/08 16:29:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:29:33 Starting bucket ops to source bucket
2021/12/08 16:29:38 Rebalance progress: 12.5
2021/12/08 16:29:43 Rebalance progress: 12.5
2021/12/08 16:29:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:29:48 Waiting for src bucket item count to get to: 200000 curr count: 9574
2021/12/08 16:29:53 Waiting for src bucket item count to get to: 200000 curr count: 25844
2021/12/08 16:29:58 Waiting for src bucket item count to get to: 200000 curr count: 25844
2021/12/08 16:30:03 Waiting for src bucket item count to get to: 200000 curr count: 42223
2021/12/08 16:30:08 Waiting for src bucket item count to get to: 200000 curr count: 42223
2021/12/08 16:30:13 Waiting for src bucket item count to get to: 200000 curr count: 58636
2021/12/08 16:30:18 Waiting for src bucket item count to get to: 200000 curr count: 58636
2021/12/08 16:30:23 Waiting for src bucket item count to get to: 200000 curr count: 58636
2021/12/08 16:30:28 Waiting for src bucket item count to get to: 200000 curr count: 74267
2021/12/08 16:30:33 Waiting for src bucket item count to get to: 200000 curr count: 74267
2021/12/08 16:30:38 Waiting for src bucket item count to get to: 200000 curr count: 90668
2021/12/08 16:30:43 Waiting for src bucket item count to get to: 200000 curr count: 90668
2021/12/08 16:30:48 Waiting for src bucket item count to get to: 200000 curr count: 107095
2021/12/08 16:30:53 Waiting for src bucket item count to get to: 200000 curr count: 107095
2021/12/08 16:30:58 Waiting for src bucket item count to get to: 200000 curr count: 123664
2021/12/08 16:31:03 Waiting for src bucket item count to get to: 200000 curr count: 123664
2021/12/08 16:31:08 Waiting for src bucket item count to get to: 200000 curr count: 139877
2021/12/08 16:31:13 Waiting for src bucket item count to get to: 200000 curr count: 139877
2021/12/08 16:31:18 Waiting for src bucket item count to get to: 200000 curr count: 156012
2021/12/08 16:31:23 Waiting for src bucket item count to get to: 200000 curr count: 156012
2021/12/08 16:31:28 Waiting for src bucket item count to get to: 200000 curr count: 171909
2021/12/08 16:31:33 Waiting for src bucket item count to get to: 200000 curr count: 171909
2021/12/08 16:31:38 Waiting for src bucket item count to get to: 200000 curr count: 188029
2021/12/08 16:31:43 Waiting for src bucket item count to get to: 200000 curr count: 188029
2021/12/08 16:31:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:31:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:31:48 Starting bucket ops to source bucket
2021/12/08 16:35:33 Waiting for dst bucket item count to get to: 0 curr count: 15694
2021/12/08 16:35:38 Waiting for dst bucket item count to get to: 0 curr count: 15694
2021/12/08 16:35:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:35:38 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:35:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:35:43 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/08 16:35:48 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/08 16:35:53 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/08 16:35:58 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/08 16:35:58 App: TestKVRebOutWithOps got un-deployed
2021/12/08 16:35:58 No eventing-con process running
2021/12/08 16:35:58 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/08 16:36:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:36:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:36:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:36:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:36:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (454.07s)
=== RUN   TestKVSwapRebWithOps
2021/12/08 16:36:22 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/08 16:36:22 App: TestKVSwapRebWithOps got un-deployed
2021/12/08 16:36:22 No eventing-con process running
2021/12/08 16:36:22 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/08 16:36:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:36:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:36:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:36:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:36:40 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:36:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/08 16:36:46 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/08 16:36:51 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/08 16:36:56 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/08 16:37:01 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/08 16:37:01 App: TestKVSwapRebWithOps got deployed
2021/12/08 16:37:01 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 16:37:01 Starting bucket ops to source bucket
2021/12/08 16:37:02 Removing node(s): [] from the cluster
2021/12/08 16:37:07 Rebalance progress: 12.5
2021/12/08 16:37:12 Rebalance progress: 12.5
2021/12/08 16:37:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:37:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 16:37:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:37:22 Rebalance progress: 12.5
2021/12/08 16:37:27 Rebalance progress: 12.5
2021/12/08 16:37:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:37:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 16:37:37 Rebalance progress: 12.5
2021/12/08 16:37:42 Rebalance progress: 12.5
2021/12/08 16:37:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:37:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:37:47 Waiting for src bucket item count to get to: 200000 curr count: 58602
2021/12/08 16:37:52 Waiting for src bucket item count to get to: 200000 curr count: 73503
2021/12/08 16:37:57 Waiting for src bucket item count to get to: 200000 curr count: 73503
2021/12/08 16:38:02 Waiting for src bucket item count to get to: 200000 curr count: 89338
2021/12/08 16:38:07 Waiting for src bucket item count to get to: 200000 curr count: 89338
2021/12/08 16:38:12 Waiting for src bucket item count to get to: 200000 curr count: 105598
2021/12/08 16:38:17 Waiting for src bucket item count to get to: 200000 curr count: 105598
2021/12/08 16:38:22 Waiting for src bucket item count to get to: 200000 curr count: 122189
2021/12/08 16:38:27 Waiting for src bucket item count to get to: 200000 curr count: 122189
2021/12/08 16:38:32 Waiting for src bucket item count to get to: 200000 curr count: 138662
2021/12/08 16:38:37 Waiting for src bucket item count to get to: 200000 curr count: 138662
2021/12/08 16:38:42 Waiting for src bucket item count to get to: 200000 curr count: 155087
2021/12/08 16:38:47 Waiting for src bucket item count to get to: 200000 curr count: 155087
2021/12/08 16:38:52 Waiting for src bucket item count to get to: 200000 curr count: 155087
2021/12/08 16:38:57 Waiting for src bucket item count to get to: 200000 curr count: 170817
2021/12/08 16:39:02 Waiting for src bucket item count to get to: 200000 curr count: 170817
2021/12/08 16:39:07 Waiting for src bucket item count to get to: 200000 curr count: 187150
2021/12/08 16:39:12 Waiting for src bucket item count to get to: 200000 curr count: 187150
2021/12/08 16:39:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/08 16:39:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/08 16:39:17 Starting bucket ops to source bucket
2021/12/08 16:43:03 Waiting for dst bucket item count to get to: 0 curr count: 16598
2021/12/08 16:43:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:43:03 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:43:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:43:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/08 16:43:14 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/08 16:43:19 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/08 16:43:24 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/08 16:43:24 App: TestKVSwapRebWithOps got un-deployed
2021/12/08 16:43:24 No eventing-con process running
2021/12/08 16:43:24 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/08 16:43:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:43:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:43:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:43:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:43:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:43:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (445.52s)
=== RUN   TestEventingRebInWithTimers
2021/12/08 16:43:47 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/08 16:43:47 App: TestEventingRebInWithTimers got un-deployed
2021/12/08 16:43:47 No eventing-con process running
2021/12/08 16:43:47 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/08 16:43:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:43:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:43:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:43:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:44:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:44:06 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:44:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 16:44:12 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/08 16:44:17 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/08 16:44:22 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/08 16:44:27 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/08 16:44:27 App: TestEventingRebInWithTimers got deployed
2021/12/08 16:44:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 16:44:27 Starting bucket ops to source bucket
2021/12/08 16:44:27 Removing node(s): [] from the cluster
2021/12/08 16:44:33 Rebalance progress: 12.5
2021/12/08 16:44:38 Rebalance progress: 12.5
2021/12/08 16:44:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:44:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:44:48 Rebalance progress: 12.5
2021/12/08 16:44:53 Rebalance progress: 12.5
2021/12/08 16:44:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:44:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:44:58 Waiting for src bucket item count to get to: 50000 curr count: 27354
2021/12/08 16:45:03 Waiting for src bucket item count to get to: 50000 curr count: 38761
2021/12/08 16:45:08 Waiting for src bucket item count to get to: 50000 curr count: 38761
2021/12/08 16:45:18 Waiting for dst bucket item count to get to: 50000 curr count: 37712
2021/12/08 16:45:23 Waiting for dst bucket item count to get to: 50000 curr count: 37712
2021/12/08 16:45:28 Waiting for dst bucket item count to get to: 50000 curr count: 49868
2021/12/08 16:45:33 Waiting for dst bucket item count to get to: 50000 curr count: 49868
2021/12/08 16:45:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 16:45:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 16:45:33 Starting bucket ops to source bucket
2021/12/08 16:46:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:46:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:46:52 Waiting for dst bucket item count to get to: 0 curr count: 49173
2021/12/08 16:46:57 Waiting for dst bucket item count to get to: 0 curr count: 49173
2021/12/08 16:47:02 Waiting for dst bucket item count to get to: 0 curr count: 35966
2021/12/08 16:47:07 Waiting for dst bucket item count to get to: 0 curr count: 35966
2021/12/08 16:47:12 Waiting for dst bucket item count to get to: 0 curr count: 22947
2021/12/08 16:47:17 Waiting for dst bucket item count to get to: 0 curr count: 22947
2021/12/08 16:47:22 Waiting for dst bucket item count to get to: 0 curr count: 2235
2021/12/08 16:47:27 Waiting for dst bucket item count to get to: 0 curr count: 2235
2021/12/08 16:47:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:47:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:47:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:47:32 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/08 16:47:37 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/08 16:47:42 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/08 16:47:47 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/08 16:47:47 App: TestEventingRebInWithTimers got un-deployed
2021/12/08 16:47:47 No eventing-con process running
2021/12/08 16:47:47 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/08 16:47:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:47:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:48:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (263.49s)
=== RUN   TestEventingRebOutWithTimers
2021/12/08 16:48:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 16:48:06 Removing node(s): [] from the cluster
2021/12/08 16:48:11 Rebalance progress: 100
2021/12/08 16:48:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/08 16:48:16 App: TestEventingRebOutWithTimers got un-deployed
2021/12/08 16:48:16 No eventing-con process running
2021/12/08 16:48:16 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/08 16:48:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:48:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:48:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:48:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:48:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:48:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:48:34 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:48:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 16:48:41 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/08 16:48:46 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/08 16:48:51 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/08 16:48:56 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/08 16:48:56 App: TestEventingRebOutWithTimers got deployed
2021/12/08 16:48:56 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:48:56 Starting bucket ops to source bucket
2021/12/08 16:49:01 Rebalance progress: 12.5
2021/12/08 16:49:06 Rebalance progress: 12.5
2021/12/08 16:49:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:49:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:49:11 Waiting for src bucket item count to get to: 50000 curr count: 19064
2021/12/08 16:49:16 Waiting for src bucket item count to get to: 50000 curr count: 19064
2021/12/08 16:49:21 Waiting for src bucket item count to get to: 50000 curr count: 29550
2021/12/08 16:49:26 Waiting for src bucket item count to get to: 50000 curr count: 29550
2021/12/08 16:49:31 Waiting for src bucket item count to get to: 50000 curr count: 41765
2021/12/08 16:49:36 Waiting for src bucket item count to get to: 50000 curr count: 41765
2021/12/08 16:49:46 Waiting for dst bucket item count to get to: 50000 curr count: 38969
2021/12/08 16:49:51 Waiting for dst bucket item count to get to: 50000 curr count: 38969
2021/12/08 16:49:56 Waiting for dst bucket item count to get to: 50000 curr count: 38969
2021/12/08 16:50:01 Waiting for dst bucket item count to get to: 50000 curr count: 47532
2021/12/08 16:50:06 Waiting for dst bucket item count to get to: 50000 curr count: 47532
2021/12/08 16:50:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 16:50:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 16:50:06 Starting bucket ops to source bucket
2021/12/08 16:51:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:51:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:51:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:51:30 Waiting for dst bucket item count to get to: 0 curr count: 47296
2021/12/08 16:51:35 Waiting for dst bucket item count to get to: 0 curr count: 47296
2021/12/08 16:51:40 Waiting for dst bucket item count to get to: 0 curr count: 35975
2021/12/08 16:51:45 Waiting for dst bucket item count to get to: 0 curr count: 35975
2021/12/08 16:51:50 Waiting for dst bucket item count to get to: 0 curr count: 11791
2021/12/08 16:51:55 Waiting for dst bucket item count to get to: 0 curr count: 11791
2021/12/08 16:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:51:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:51:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:52:00 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/08 16:52:05 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/08 16:52:10 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/08 16:52:15 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/08 16:52:16 App: TestEventingRebOutWithTimers got un-deployed
2021/12/08 16:52:16 No eventing-con process running
2021/12/08 16:52:16 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/08 16:52:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:52:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:52:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:52:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (268.40s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/08 16:52:39 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/08 16:52:39 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/08 16:52:39 No eventing-con process running
2021/12/08 16:52:39 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/08 16:52:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:52:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:52:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:52:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:52:58 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:52:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 16:53:04 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/08 16:53:09 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/08 16:53:14 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/08 16:53:19 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/08 16:53:19 App: TestEventingSwapRebWithTimers got deployed
2021/12/08 16:53:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 16:53:19 Starting bucket ops to source bucket
2021/12/08 16:53:19 Removing node(s): [] from the cluster
2021/12/08 16:53:24 Rebalance progress: 12.5
2021/12/08 16:53:29 Rebalance progress: 12.5
2021/12/08 16:53:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:53:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:53:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:53:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 16:53:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:53:40 Rebalance progress: 12.5
2021/12/08 16:53:45 Rebalance progress: 12.5
2021/12/08 16:53:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:53:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:53:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 16:53:55 Rebalance progress: 12.5
2021/12/08 16:54:00 Rebalance progress: 12.5
2021/12/08 16:54:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:54:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:54:05 Waiting for src bucket item count to get to: 50000 curr count: 46920
2021/12/08 16:54:10 Waiting for src bucket item count to get to: 50000 curr count: 46920
2021/12/08 16:54:20 Waiting for dst bucket item count to get to: 50000 curr count: 47925
2021/12/08 16:54:25 Waiting for dst bucket item count to get to: 50000 curr count: 47925
2021/12/08 16:54:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 16:54:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 16:54:25 Starting bucket ops to source bucket
2021/12/08 16:55:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:55:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:55:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:55:50 Waiting for dst bucket item count to get to: 0 curr count: 40521
2021/12/08 16:55:55 Waiting for dst bucket item count to get to: 0 curr count: 40521
2021/12/08 16:56:00 Waiting for dst bucket item count to get to: 0 curr count: 24765
2021/12/08 16:56:05 Waiting for dst bucket item count to get to: 0 curr count: 24765
2021/12/08 16:56:10 Waiting for dst bucket item count to get to: 0 curr count: 11546
2021/12/08 16:56:15 Waiting for dst bucket item count to get to: 0 curr count: 11546
2021/12/08 16:56:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:56:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 16:56:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 16:56:20 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/08 16:56:25 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/08 16:56:30 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/08 16:56:35 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/08 16:56:35 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/08 16:56:35 No eventing-con process running
2021/12/08 16:56:35 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/08 16:56:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:56:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:56:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:56:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:56:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (259.68s)
=== RUN   TestKVRebInWithTimers
2021/12/08 16:56:59 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/08 16:56:59 App: TestKVRebInWithTimers got un-deployed
2021/12/08 16:56:59 No eventing-con process running
2021/12/08 16:56:59 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/08 16:57:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:57:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 16:57:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:57:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:57:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 16:57:17 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 16:57:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 16:57:23 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/08 16:57:28 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/08 16:57:33 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/08 16:57:38 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/08 16:57:38 App: TestKVRebInWithTimers got deployed
2021/12/08 16:57:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 16:57:38 Starting bucket ops to source bucket
2021/12/08 16:57:39 Removing node(s): [] from the cluster
2021/12/08 16:57:44 Rebalance progress: 12.5
2021/12/08 16:57:49 Rebalance progress: 12.5
2021/12/08 16:57:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:57:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:57:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 16:57:59 Rebalance progress: 12.5
2021/12/08 16:58:04 Rebalance progress: 12.5
2021/12/08 16:58:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 16:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 16:58:09 Waiting for src bucket item count to get to: 50000 curr count: 25469
2021/12/08 16:58:14 Waiting for src bucket item count to get to: 50000 curr count: 35572
2021/12/08 16:58:19 Waiting for src bucket item count to get to: 50000 curr count: 35572
2021/12/08 16:58:24 Waiting for src bucket item count to get to: 50000 curr count: 48308
2021/12/08 16:58:29 Waiting for src bucket item count to get to: 50000 curr count: 48308
2021/12/08 16:58:39 Waiting for dst bucket item count to get to: 50000 curr count: 48591
2021/12/08 16:58:44 Waiting for dst bucket item count to get to: 50000 curr count: 48591
2021/12/08 16:58:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 16:58:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 16:58:44 Starting bucket ops to source bucket
2021/12/08 16:59:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 16:59:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:00:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:00:08 Waiting for dst bucket item count to get to: 0 curr count: 39940
2021/12/08 17:00:13 Waiting for dst bucket item count to get to: 0 curr count: 39940
2021/12/08 17:00:18 Waiting for dst bucket item count to get to: 0 curr count: 23519
2021/12/08 17:00:23 Waiting for dst bucket item count to get to: 0 curr count: 23519
2021/12/08 17:00:28 Waiting for dst bucket item count to get to: 0 curr count: 9379
2021/12/08 17:00:33 Waiting for dst bucket item count to get to: 0 curr count: 9379
2021/12/08 17:00:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:00:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 17:00:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:00:38 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/08 17:00:43 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/08 17:00:48 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/08 17:00:53 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/08 17:00:53 App: TestKVRebInWithTimers got un-deployed
2021/12/08 17:00:53 No eventing-con process running
2021/12/08 17:00:53 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/08 17:00:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:01:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:01:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:01:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:01:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:01:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.23s)
=== RUN   TestKVRebOutWithTimers
2021/12/08 17:01:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:01:12 Removing node(s): [] from the cluster
2021/12/08 17:01:17 Rebalance progress: 100
2021/12/08 17:01:22 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/08 17:01:22 App: TestKVRebOutWithTimers got un-deployed
2021/12/08 17:01:22 No eventing-con process running
2021/12/08 17:01:22 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/08 17:01:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:01:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:01:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:01:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:01:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:01:41 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:01:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 17:01:47 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/08 17:01:52 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/08 17:01:57 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/08 17:02:02 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/08 17:02:02 App: TestKVRebOutWithTimers got deployed
2021/12/08 17:02:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:02:02 Starting bucket ops to source bucket
2021/12/08 17:02:07 Rebalance progress: 12.5
2021/12/08 17:02:12 Rebalance progress: 12.5
2021/12/08 17:02:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:02:17 Waiting for src bucket item count to get to: 50000 curr count: 10080
2021/12/08 17:02:22 Waiting for src bucket item count to get to: 50000 curr count: 24410
2021/12/08 17:02:27 Waiting for src bucket item count to get to: 50000 curr count: 24410
2021/12/08 17:02:32 Waiting for src bucket item count to get to: 50000 curr count: 36195
2021/12/08 17:02:37 Waiting for src bucket item count to get to: 50000 curr count: 36195
2021/12/08 17:02:42 Waiting for src bucket item count to get to: 50000 curr count: 46275
2021/12/08 17:02:47 Waiting for src bucket item count to get to: 50000 curr count: 46275
2021/12/08 17:02:52 Waiting for src bucket item count to get to: 50000 curr count: 46275
2021/12/08 17:03:02 Waiting for dst bucket item count to get to: 50000 curr count: 36127
2021/12/08 17:03:07 Waiting for dst bucket item count to get to: 50000 curr count: 36127
2021/12/08 17:03:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:03:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:03:07 Starting bucket ops to source bucket
2021/12/08 17:04:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:04:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:04:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:04:32 Waiting for dst bucket item count to get to: 0 curr count: 49934
2021/12/08 17:04:37 Waiting for dst bucket item count to get to: 0 curr count: 49934
2021/12/08 17:04:42 Waiting for dst bucket item count to get to: 0 curr count: 34665
2021/12/08 17:04:47 Waiting for dst bucket item count to get to: 0 curr count: 34665
2021/12/08 17:04:52 Waiting for dst bucket item count to get to: 0 curr count: 19155
2021/12/08 17:04:57 Waiting for dst bucket item count to get to: 0 curr count: 19155
2021/12/08 17:05:02 Waiting for dst bucket item count to get to: 0 curr count: 6604
2021/12/08 17:05:07 Waiting for dst bucket item count to get to: 0 curr count: 6604
2021/12/08 17:05:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:05:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 17:05:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:05:12 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/08 17:05:17 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/08 17:05:22 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/08 17:05:27 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/08 17:05:27 App: TestKVRebOutWithTimers got un-deployed
2021/12/08 17:05:27 No eventing-con process running
2021/12/08 17:05:27 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/08 17:05:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:05:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:05:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:05:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:05:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (273.58s)
=== RUN   TestKVSwapRebWithTimers
2021/12/08 17:05:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:05:51 App: TestKVSwapRebWithTimers got un-deployed
2021/12/08 17:05:51 No eventing-con process running
2021/12/08 17:05:51 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/08 17:05:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:05:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:06:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:06:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:06:09 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:06:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/08 17:06:15 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/08 17:06:20 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/08 17:06:25 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/08 17:06:30 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/08 17:06:30 App: TestKVSwapRebWithTimers got deployed
2021/12/08 17:06:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:06:30 Starting bucket ops to source bucket
2021/12/08 17:06:31 Removing node(s): [] from the cluster
2021/12/08 17:06:36 Rebalance progress: 12.5
2021/12/08 17:06:41 Rebalance progress: 12.5
2021/12/08 17:06:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:06:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:06:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:06:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:06:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:06:51 Rebalance progress: 12.5
2021/12/08 17:06:56 Rebalance progress: 12.5
2021/12/08 17:07:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:07:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:07:06 Rebalance progress: 12.5
2021/12/08 17:07:11 Rebalance progress: 12.5
2021/12/08 17:07:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:07:16 Waiting for src bucket item count to get to: 50000 curr count: 44815
2021/12/08 17:07:26 Waiting for dst bucket item count to get to: 50000 curr count: 43065
2021/12/08 17:07:31 Waiting for dst bucket item count to get to: 50000 curr count: 43065
2021/12/08 17:07:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:07:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:07:31 Starting bucket ops to source bucket
2021/12/08 17:08:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:08:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:08:50 Waiting for dst bucket item count to get to: 0 curr count: 48306
2021/12/08 17:08:55 Waiting for dst bucket item count to get to: 0 curr count: 48306
2021/12/08 17:09:00 Waiting for dst bucket item count to get to: 0 curr count: 38009
2021/12/08 17:09:05 Waiting for dst bucket item count to get to: 0 curr count: 38009
2021/12/08 17:09:10 Waiting for dst bucket item count to get to: 0 curr count: 15596
2021/12/08 17:09:15 Waiting for dst bucket item count to get to: 0 curr count: 15596
2021/12/08 17:09:20 Waiting for dst bucket item count to get to: 0 curr count: 3227
2021/12/08 17:09:25 Waiting for dst bucket item count to get to: 0 curr count: 3227
2021/12/08 17:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:09:25 Post DELETE ops. Expected item count: 0 got 0
2021/12/08 17:09:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:09:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:09:36 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:09:41 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:09:46 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:09:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/08 17:09:51 App: TestKVSwapRebWithTimers got un-deployed
2021/12/08 17:09:51 No eventing-con process running
2021/12/08 17:09:51 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/08 17:09:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:09:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:09:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:09:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:10:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (263.71s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/08 17:10:14 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:10:14 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/08 17:10:14 No eventing-con process running
2021/12/08 17:10:14 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/08 17:10:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:10:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:10:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:10:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:10:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:10:33 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:10:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 17:10:39 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/08 17:10:44 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/08 17:10:49 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/08 17:10:54 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/08 17:10:54 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/08 17:10:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:10:54 Starting bucket ops to source bucket
2021/12/08 17:10:54 Removing node(s): [] from the cluster
2021/12/08 17:10:59 Rebalance progress: 12.5
2021/12/08 17:11:04 Rebalance progress: 12.5
2021/12/08 17:11:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:11:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:11:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:11:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:11:15 Rebalance progress: 12.5
2021/12/08 17:11:20 Rebalance progress: 12.5
2021/12/08 17:11:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:11:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:11:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:11:25 Waiting for src bucket item count to get to: 50000 curr count: 33826
2021/12/08 17:11:30 Waiting for src bucket item count to get to: 50000 curr count: 33826
2021/12/08 17:11:35 Waiting for src bucket item count to get to: 50000 curr count: 44441
2021/12/08 17:11:40 Waiting for src bucket item count to get to: 50000 curr count: 44441
2021/12/08 17:11:50 Waiting for dst bucket item count to get to: 50000 curr count: 28270
2021/12/08 17:11:55 Waiting for dst bucket item count to get to: 50000 curr count: 28270
2021/12/08 17:12:00 Waiting for dst bucket item count to get to: 50000 curr count: 37039
2021/12/08 17:12:05 Waiting for dst bucket item count to get to: 50000 curr count: 37039
2021/12/08 17:12:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:12:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:12:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:12:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:15 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/08 17:12:35 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/08 17:12:35 No eventing-con process running
2021/12/08 17:12:35 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/08 17:12:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:12:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:12:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:12:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:12:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:12:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (174.24s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/08 17:13:03 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:13:04 Removing node(s): [] from the cluster
2021/12/08 17:13:09 Rebalance progress: 100
2021/12/08 17:13:14 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:13:14 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/08 17:13:14 No eventing-con process running
2021/12/08 17:13:14 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/08 17:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:13:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:13:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:13:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:13:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:13:32 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:13:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 17:13:38 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/08 17:13:43 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/08 17:13:48 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/08 17:13:53 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/08 17:13:53 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/08 17:13:53 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:13:53 Starting bucket ops to source bucket
2021/12/08 17:13:59 Rebalance progress: 12.5
2021/12/08 17:14:04 Rebalance progress: 12.5
2021/12/08 17:14:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:14:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:14:09 Waiting for src bucket item count to get to: 50000 curr count: 7268
2021/12/08 17:14:14 Waiting for src bucket item count to get to: 50000 curr count: 21380
2021/12/08 17:14:19 Waiting for src bucket item count to get to: 50000 curr count: 21380
2021/12/08 17:14:24 Waiting for src bucket item count to get to: 50000 curr count: 34754
2021/12/08 17:14:29 Waiting for src bucket item count to get to: 50000 curr count: 34754
2021/12/08 17:14:34 Waiting for src bucket item count to get to: 50000 curr count: 44999
2021/12/08 17:14:39 Waiting for src bucket item count to get to: 50000 curr count: 44999
2021/12/08 17:14:44 Waiting for src bucket item count to get to: 50000 curr count: 44999
2021/12/08 17:14:54 Waiting for dst bucket item count to get to: 50000 curr count: 32158
2021/12/08 17:14:59 Waiting for dst bucket item count to get to: 50000 curr count: 32158
2021/12/08 17:15:04 Waiting for dst bucket item count to get to: 50000 curr count: 40550
2021/12/08 17:15:09 Waiting for dst bucket item count to get to: 50000 curr count: 40550
2021/12/08 17:15:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:15:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:15:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:15:14 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:15:19 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:15:24 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:15:29 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:15:34 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/08 17:15:34 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/08 17:15:34 No eventing-con process running
2021/12/08 17:15:34 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/08 17:15:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:15:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:15:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:15:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:15:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:15:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:15:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:15:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.13s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/08 17:16:08 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:16:08 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/08 17:16:08 No eventing-con process running
2021/12/08 17:16:08 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/08 17:16:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:16:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:16:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:16:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:16:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:16:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:16:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:16:26 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:16:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 17:16:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/08 17:16:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/08 17:16:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/08 17:16:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/08 17:16:47 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/08 17:16:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:16:47 Starting bucket ops to source bucket
2021/12/08 17:16:48 Removing node(s): [] from the cluster
2021/12/08 17:16:53 Rebalance progress: 12.5
2021/12/08 17:16:58 Rebalance progress: 12.5
2021/12/08 17:17:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:17:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 17:17:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:17:08 Rebalance progress: 12.5
2021/12/08 17:17:13 Rebalance progress: 12.5
2021/12/08 17:17:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:17:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:17:24 Rebalance progress: 12.5
2021/12/08 17:17:29 Rebalance progress: 12.5
2021/12/08 17:17:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:17:39 Waiting for dst bucket item count to get to: 50000 curr count: 26222
2021/12/08 17:17:44 Waiting for dst bucket item count to get to: 50000 curr count: 26222
2021/12/08 17:17:49 Waiting for dst bucket item count to get to: 50000 curr count: 39241
2021/12/08 17:17:54 Waiting for dst bucket item count to get to: 50000 curr count: 39241
2021/12/08 17:17:59 Waiting for dst bucket item count to get to: 50000 curr count: 46385
2021/12/08 17:18:04 Waiting for dst bucket item count to get to: 50000 curr count: 46385
2021/12/08 17:18:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:18:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:18:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:34 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/08 17:18:34 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/08 17:18:34 No eventing-con process running
2021/12/08 17:18:34 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/08 17:18:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:18:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:18:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:18:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:18:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:18:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.86s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/08 17:19:07 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:19:07 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/08 17:19:07 No eventing-con process running
2021/12/08 17:19:07 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/08 17:19:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:19:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:19:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:19:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:19:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:19:26 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:19:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 17:19:32 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/08 17:19:37 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/08 17:19:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/08 17:19:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/08 17:19:47 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/08 17:19:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:19:47 Starting bucket ops to source bucket
2021/12/08 17:19:48 Removing node(s): [] from the cluster
2021/12/08 17:19:53 Rebalance progress: 12.5
2021/12/08 17:19:58 Rebalance progress: 12.5
2021/12/08 17:20:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:20:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:20:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:20:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:20:08 Rebalance progress: 12.5
2021/12/08 17:20:13 Rebalance progress: 12.5
2021/12/08 17:20:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:20:18 Waiting for src bucket item count to get to: 50000 curr count: 28353
2021/12/08 17:20:23 Waiting for src bucket item count to get to: 50000 curr count: 39538
2021/12/08 17:20:28 Waiting for src bucket item count to get to: 50000 curr count: 39538
2021/12/08 17:20:38 Waiting for dst bucket item count to get to: 50000 curr count: 23853
2021/12/08 17:20:43 Waiting for dst bucket item count to get to: 50000 curr count: 23853
2021/12/08 17:20:48 Waiting for dst bucket item count to get to: 50000 curr count: 37609
2021/12/08 17:20:53 Waiting for dst bucket item count to get to: 50000 curr count: 37609
2021/12/08 17:20:58 Waiting for dst bucket item count to get to: 50000 curr count: 44837
2021/12/08 17:21:03 Waiting for dst bucket item count to get to: 50000 curr count: 44837
2021/12/08 17:21:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:21:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:21:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:21:08 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:23 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:33 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/08 17:21:33 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/08 17:21:33 No eventing-con process running
2021/12/08 17:21:33 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/08 17:21:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:21:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:21:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:21:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:21:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:21:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:21:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.37s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/08 17:22:02 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:22:02 Removing node(s): [] from the cluster
2021/12/08 17:22:07 Rebalance progress: 100
2021/12/08 17:22:12 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:22:12 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/08 17:22:12 No eventing-con process running
2021/12/08 17:22:12 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/08 17:22:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:22:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:22:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:22:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:22:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:22:31 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:22:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 17:22:37 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/08 17:22:42 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/08 17:22:47 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/08 17:22:52 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/08 17:22:52 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/08 17:22:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:22:52 Starting bucket ops to source bucket
2021/12/08 17:22:57 Rebalance progress: 12.5
2021/12/08 17:23:02 Rebalance progress: 12.5
2021/12/08 17:23:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:23:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:23:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:23:07 Waiting for src bucket item count to get to: 50000 curr count: 9341
2021/12/08 17:23:12 Waiting for src bucket item count to get to: 50000 curr count: 23354
2021/12/08 17:23:17 Waiting for src bucket item count to get to: 50000 curr count: 23354
2021/12/08 17:23:22 Waiting for src bucket item count to get to: 50000 curr count: 36729
2021/12/08 17:23:27 Waiting for src bucket item count to get to: 50000 curr count: 36729
2021/12/08 17:23:32 Waiting for src bucket item count to get to: 50000 curr count: 36729
2021/12/08 17:23:37 Waiting for src bucket item count to get to: 50000 curr count: 45955
2021/12/08 17:23:42 Waiting for src bucket item count to get to: 50000 curr count: 45955
2021/12/08 17:23:52 Waiting for dst bucket item count to get to: 50000 curr count: 32768
2021/12/08 17:23:57 Waiting for dst bucket item count to get to: 50000 curr count: 32768
2021/12/08 17:24:02 Waiting for dst bucket item count to get to: 50000 curr count: 47076
2021/12/08 17:24:07 Waiting for dst bucket item count to get to: 50000 curr count: 47076
2021/12/08 17:24:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:24:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:24:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:24:12 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:17 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:22 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:27 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:32 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:37 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/08 17:24:37 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/08 17:24:37 No eventing-con process running
2021/12/08 17:24:37 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/08 17:24:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:24:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:24:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:24:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:24:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:24:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:25:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:25:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.26s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/08 17:25:16 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:25:16 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/08 17:25:16 No eventing-con process running
2021/12/08 17:25:16 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/08 17:25:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:25:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:25:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:25:35 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:25:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 17:25:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/08 17:25:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/08 17:25:51 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/08 17:25:56 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/08 17:25:56 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/08 17:25:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:25:56 Starting bucket ops to source bucket
2021/12/08 17:25:56 Removing node(s): [] from the cluster
2021/12/08 17:26:01 Rebalance progress: 12.5
2021/12/08 17:26:06 Rebalance progress: 12.5
2021/12/08 17:26:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:26:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:26:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:26:17 Rebalance progress: 12.5
2021/12/08 17:26:22 Rebalance progress: 12.5
2021/12/08 17:26:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:26:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:26:32 Rebalance progress: 12.5
2021/12/08 17:26:37 Rebalance progress: 12.5
2021/12/08 17:26:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:26:47 Waiting for dst bucket item count to get to: 50000 curr count: 29511
2021/12/08 17:26:52 Waiting for dst bucket item count to get to: 50000 curr count: 29511
2021/12/08 17:26:57 Waiting for dst bucket item count to get to: 50000 curr count: 36239
2021/12/08 17:27:02 Waiting for dst bucket item count to get to: 50000 curr count: 36239
2021/12/08 17:27:07 Waiting for dst bucket item count to get to: 50000 curr count: 47423
2021/12/08 17:27:12 Waiting for dst bucket item count to get to: 50000 curr count: 47423
2021/12/08 17:27:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:27:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:27:17 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:22 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:43 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/08 17:27:43 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/08 17:27:43 No eventing-con process running
2021/12/08 17:27:43 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/08 17:27:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:27:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:27:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:27:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:28:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:28:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:28:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.20s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/08 17:28:16 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:28:16 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/08 17:28:16 No eventing-con process running
2021/12/08 17:28:16 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/08 17:28:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:28:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:28:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:28:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:28:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:28:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:28:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:28:35 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:28:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 17:28:41 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/08 17:28:46 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/08 17:28:51 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/08 17:28:56 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/08 17:28:56 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/08 17:28:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:28:56 Starting bucket ops to source bucket
2021/12/08 17:28:56 Removing node(s): [] from the cluster
2021/12/08 17:29:01 Rebalance progress: 12.5
2021/12/08 17:29:06 Rebalance progress: 12.5
2021/12/08 17:29:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:29:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:29:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:29:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:29:16 Rebalance progress: 12.5
2021/12/08 17:29:21 Rebalance progress: 12.5
2021/12/08 17:29:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:29:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:29:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:29:27 Waiting for src bucket item count to get to: 50000 curr count: 28213
2021/12/08 17:29:32 Waiting for src bucket item count to get to: 50000 curr count: 39815
2021/12/08 17:29:37 Waiting for src bucket item count to get to: 50000 curr count: 39815
2021/12/08 17:29:47 Waiting for dst bucket item count to get to: 50000 curr count: 38177
2021/12/08 17:29:52 Waiting for dst bucket item count to get to: 50000 curr count: 38177
2021/12/08 17:29:57 Waiting for dst bucket item count to get to: 50000 curr count: 47018
2021/12/08 17:30:02 Waiting for dst bucket item count to get to: 50000 curr count: 47018
2021/12/08 17:30:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:30:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:30:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:30:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:12 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:17 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:22 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:27 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:32 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/08 17:30:32 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/08 17:30:32 No eventing-con process running
2021/12/08 17:30:32 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/08 17:30:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:30:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:30:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:30:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:30:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:31:00 Waiting for dst bucket item count to get to: 0 curr count: 25972
2021/12/08 17:31:05 Waiting for dst bucket item count to get to: 0 curr count: 25972
2021/12/08 17:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.16s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/08 17:31:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:31:11 Removing node(s): [] from the cluster
2021/12/08 17:31:16 Rebalance progress: 100
2021/12/08 17:31:21 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:31:21 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/08 17:31:21 No eventing-con process running
2021/12/08 17:31:21 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/08 17:31:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:31:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:31:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:31:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:31:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:31:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:31:39 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:31:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 17:31:46 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/08 17:31:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/08 17:31:56 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/08 17:32:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/08 17:32:01 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/08 17:32:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:32:01 Starting bucket ops to source bucket
2021/12/08 17:32:06 Rebalance progress: 12.5
2021/12/08 17:32:11 Rebalance progress: 12.5
2021/12/08 17:32:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:32:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:32:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:32:16 Waiting for src bucket item count to get to: 50000 curr count: 11577
2021/12/08 17:32:21 Waiting for src bucket item count to get to: 50000 curr count: 25504
2021/12/08 17:32:26 Waiting for src bucket item count to get to: 50000 curr count: 25504
2021/12/08 17:32:31 Waiting for src bucket item count to get to: 50000 curr count: 36022
2021/12/08 17:32:36 Waiting for src bucket item count to get to: 50000 curr count: 36022
2021/12/08 17:32:41 Waiting for src bucket item count to get to: 50000 curr count: 47027
2021/12/08 17:32:46 Waiting for src bucket item count to get to: 50000 curr count: 47027
2021/12/08 17:32:51 Waiting for src bucket item count to get to: 50000 curr count: 47027
2021/12/08 17:33:01 Waiting for dst bucket item count to get to: 50000 curr count: 36445
2021/12/08 17:33:06 Waiting for dst bucket item count to get to: 50000 curr count: 36445
2021/12/08 17:33:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:33:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:33:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:33:11 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:16 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:21 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:26 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:31 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:36 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/08 17:33:36 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/08 17:33:36 No eventing-con process running
2021/12/08 17:33:36 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/08 17:33:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:33:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:33:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:33:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:33:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:33:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.33s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/08 17:34:05 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:34:05 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/08 17:34:05 No eventing-con process running
2021/12/08 17:34:05 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/08 17:34:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:34:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:34:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:34:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:34:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:34:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:34:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:34:23 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:34:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 17:34:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/08 17:34:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/08 17:34:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/08 17:34:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/08 17:34:44 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/08 17:34:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/08 17:34:44 Starting bucket ops to source bucket
2021/12/08 17:34:45 Removing node(s): [] from the cluster
2021/12/08 17:34:50 Rebalance progress: 12.5
2021/12/08 17:34:55 Rebalance progress: 12.5
2021/12/08 17:35:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:35:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 17:35:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:35:06 Rebalance progress: 12.5
2021/12/08 17:35:11 Rebalance progress: 12.5
2021/12/08 17:35:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:35:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:35:21 Rebalance progress: 12.5
2021/12/08 17:35:26 Rebalance progress: 12.5
2021/12/08 17:35:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:35:36 Waiting for dst bucket item count to get to: 50000 curr count: 39539
2021/12/08 17:35:41 Waiting for dst bucket item count to get to: 50000 curr count: 39539
2021/12/08 17:35:46 Waiting for dst bucket item count to get to: 50000 curr count: 47769
2021/12/08 17:35:51 Waiting for dst bucket item count to get to: 50000 curr count: 47769
2021/12/08 17:35:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:35:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:35:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:35:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:01 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:21 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/08 17:36:21 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/08 17:36:21 No eventing-con process running
2021/12/08 17:36:21 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/08 17:36:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:36:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:36:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:36:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:36:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:36:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.83s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/08 17:36:55 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:36:55 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/08 17:36:55 No eventing-con process running
2021/12/08 17:36:55 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/08 17:36:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:37:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:37:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:37:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:37:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:37:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:37:13 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:37:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/08 17:37:19 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/08 17:37:24 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/08 17:37:29 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/08 17:37:34 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/08 17:37:34 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/08 17:37:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:37:34 Starting bucket ops to source bucket
2021/12/08 17:37:35 Removing node(s): [] from the cluster
2021/12/08 17:37:40 Rebalance progress: 12.5
2021/12/08 17:37:45 Rebalance progress: 12.5
2021/12/08 17:37:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:37:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:37:55 Rebalance progress: 12.5
2021/12/08 17:38:00 Rebalance progress: 12.5
2021/12/08 17:38:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:38:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:38:05 Waiting for src bucket item count to get to: 50000 curr count: 29499
2021/12/08 17:38:10 Waiting for src bucket item count to get to: 50000 curr count: 29499
2021/12/08 17:38:15 Waiting for src bucket item count to get to: 50000 curr count: 41478
2021/12/08 17:38:20 Waiting for src bucket item count to get to: 50000 curr count: 41478
2021/12/08 17:38:30 Waiting for dst bucket item count to get to: 50000 curr count: 37815
2021/12/08 17:38:35 Waiting for dst bucket item count to get to: 50000 curr count: 37815
2021/12/08 17:38:40 Waiting for dst bucket item count to get to: 50000 curr count: 46002
2021/12/08 17:38:40 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:38:40 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:38:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:38:45 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:38:50 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:38:55 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:39:00 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:39:05 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:39:10 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/08 17:39:10 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/08 17:39:10 No eventing-con process running
2021/12/08 17:39:10 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/08 17:39:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:39:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:39:22 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:39:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:39:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:39:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:39:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:39:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:39:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (174.40s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/08 17:39:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:39:44 Removing node(s): [] from the cluster
2021/12/08 17:39:49 Rebalance progress: 100
2021/12/08 17:39:54 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:39:54 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/08 17:39:54 No eventing-con process running
2021/12/08 17:39:54 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/08 17:39:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:40:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:40:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:40:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:40:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:40:13 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:40:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/08 17:40:19 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/08 17:40:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/08 17:40:29 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/08 17:40:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/08 17:40:34 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/08 17:40:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:40:34 Starting bucket ops to source bucket
2021/12/08 17:40:39 Rebalance progress: 12.5
2021/12/08 17:40:44 Rebalance progress: 12.5
2021/12/08 17:40:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:40:49 Waiting for src bucket item count to get to: 50000 curr count: 6857
2021/12/08 17:40:54 Waiting for src bucket item count to get to: 50000 curr count: 19823
2021/12/08 17:40:59 Waiting for src bucket item count to get to: 50000 curr count: 19823
2021/12/08 17:41:04 Waiting for src bucket item count to get to: 50000 curr count: 30398
2021/12/08 17:41:09 Waiting for src bucket item count to get to: 50000 curr count: 30398
2021/12/08 17:41:14 Waiting for src bucket item count to get to: 50000 curr count: 30398
2021/12/08 17:41:19 Waiting for src bucket item count to get to: 50000 curr count: 42147
2021/12/08 17:41:24 Waiting for src bucket item count to get to: 50000 curr count: 42147
2021/12/08 17:41:34 Waiting for dst bucket item count to get to: 50000 curr count: 41419
2021/12/08 17:41:39 Waiting for dst bucket item count to get to: 50000 curr count: 41419
2021/12/08 17:41:44 Waiting for dst bucket item count to get to: 50000 curr count: 49583
2021/12/08 17:41:49 Waiting for dst bucket item count to get to: 50000 curr count: 49583
2021/12/08 17:41:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:41:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:41:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:41:54 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:41:59 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:42:04 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:42:09 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:42:14 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:42:19 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/08 17:42:20 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/08 17:42:20 No eventing-con process running
2021/12/08 17:42:20 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/08 17:42:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:42:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:42:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:42:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:42:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.12s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/08 17:42:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:42:58 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/08 17:42:58 No eventing-con process running
2021/12/08 17:42:58 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/08 17:43:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:43:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:43:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:43:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:43:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:43:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:43:17 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:43:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/08 17:43:23 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/08 17:43:28 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/08 17:43:33 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/08 17:43:38 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/08 17:43:38 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/08 17:43:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:43:38 Starting bucket ops to source bucket
2021/12/08 17:43:38 Removing node(s): [] from the cluster
2021/12/08 17:43:43 Rebalance progress: 12.5
2021/12/08 17:43:48 Rebalance progress: 12.5
2021/12/08 17:43:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:43:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:43:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:43:59 Rebalance progress: 12.5
2021/12/08 17:44:04 Rebalance progress: 12.5
2021/12/08 17:44:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:44:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:44:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:44:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:44:14 Rebalance progress: 12.5
2021/12/08 17:44:19 Rebalance progress: 12.5
2021/12/08 17:44:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:44:24 Waiting for src bucket item count to get to: 50000 curr count: 47998
2021/12/08 17:44:29 Waiting for src bucket item count to get to: 50000 curr count: 47998
2021/12/08 17:44:39 Waiting for dst bucket item count to get to: 50000 curr count: 45435
2021/12/08 17:44:44 Waiting for dst bucket item count to get to: 50000 curr count: 45435
2021/12/08 17:44:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/08 17:44:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/08 17:44:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:44:49 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:44:54 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:45:00 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:45:05 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:45:10 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:45:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/08 17:45:15 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/08 17:45:15 No eventing-con process running
2021/12/08 17:45:15 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/08 17:45:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:45:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:45:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:45:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:45:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/08 17:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.16s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/08 17:45:48 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:45:48 App: bucket_op_on_update got un-deployed
2021/12/08 17:45:48 No eventing-con process running
2021/12/08 17:45:48 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:45:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:45:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:45:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:46:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:46:07 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:46:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 17:46:13 Starting bucket ops to source bucket
2021/12/08 17:46:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:46:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:46:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:46:28 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:46:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:46:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:46:33 Rebalance progress: 12.5
2021/12/08 17:46:38 Rebalance progress: 12.5
2021/12/08 17:46:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:46:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:46:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:46:49 Rebalance progress: 12.5
2021/12/08 17:46:54 Rebalance progress: 12.5
2021/12/08 17:46:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:46:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 17:46:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:47:04 Rebalance progress: 12.5
2021/12/08 17:47:09 Rebalance progress: 12.5
2021/12/08 17:47:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:47:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:47:19 Rebalance progress: 12.5
2021/12/08 17:47:24 Rebalance progress: 12.5
2021/12/08 17:47:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:47:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:47:34 Rebalance progress: 12.5
2021/12/08 17:47:39 Rebalance progress: 12.5
2021/12/08 17:47:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:47:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 17:47:49 Rebalance progress: 12.5
2021/12/08 17:47:54 Rebalance progress: 12.5
2021/12/08 17:47:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:47:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:47:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:48:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:48:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:48:30 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:48:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 17:48:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:48:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:48:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10625
2021/12/08 17:48:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10625
2021/12/08 17:48:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:48:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:48:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:48:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10625
2021/12/08 17:48:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (199.83s)
=== RUN   TestKVForceFailover
2021/12/08 17:49:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:49:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:49:13 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:49:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:49:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:49:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:49:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:49:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:49:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:49:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:49:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:49:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:49:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:49:32 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:49:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 17:49:33 Starting bucket ops to source bucket
2021/12/08 17:49:38 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:49:43 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:49:48 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:49:53 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:49:53 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:49:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:49:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:49:53 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/08 17:50:13 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/08 17:50:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/08 17:50:18 Rebalance progress: 12.5
2021/12/08 17:50:23 Rebalance progress: 12.5
2021/12/08 17:50:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:50:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:50:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:50:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/08 17:50:33 Rebalance progress: 12.5
2021/12/08 17:50:38 Rebalance progress: 12.5
2021/12/08 17:50:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:50:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:50:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:50:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:50:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:50:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:50:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:51:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:51:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:51:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:51:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:51:14 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:51:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 17:51:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:51:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:51:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7031
2021/12/08 17:51:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:51:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:51:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:51:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7031
2021/12/08 17:51:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.14s)
=== RUN   TestKVHardFailoverRecovery
2021/12/08 17:51:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:51:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:51:47 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:51:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:51:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:51:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:51:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:51:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:52:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:52:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:52:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 17:52:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 17:52:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:52:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:52:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:52:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:52:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:52:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:52:33 Rebalance progress: 12.5
2021/12/08 17:52:38 Rebalance progress: 12.5
2021/12/08 17:52:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:52:43 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/08 17:53:43 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/08 17:53:48 Rebalance progress: 12.5
2021/12/08 17:53:53 Rebalance progress: 12.5
2021/12/08 17:53:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:53:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:54:03 Rebalance progress: 12.5
2021/12/08 17:54:08 Rebalance progress: 12.5
2021/12/08 17:54:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:54:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:54:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:54:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:54:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:54:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:54:43 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:54:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 17:54:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:54:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:54:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12076
2021/12/08 17:54:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:54:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:55:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12076
2021/12/08 17:55:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.51s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/08 17:55:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:55:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:55:17 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:55:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:55:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:55:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:55:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:55:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:55:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:55:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 17:55:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 17:55:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:55:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:55:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:55:56 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:55:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:55:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:55:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:55:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 17:55:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:56:02 Rebalance progress: 12.5
2021/12/08 17:56:07 Rebalance progress: 12.5
2021/12/08 17:56:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:56:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/08 17:56:17 Rebalance progress: 12.5
2021/12/08 17:56:22 Rebalance progress: 12.5
2021/12/08 17:56:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:56:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:56:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:56:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:56:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:56:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:56:58 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:56:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 17:57:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:57:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:57:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:57:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:57:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:57:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/08 17:57:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.69s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/08 17:57:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:57:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:57:26 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:57:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:57:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:57:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:57:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:57:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:57:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:57:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:57:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:57:40 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 17:57:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 17:57:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 17:57:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:57:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:58:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 17:58:01 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:58:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 17:58:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 17:58:07 Rebalance progress: 12.5
2021/12/08 17:58:12 Rebalance progress: 12.5
2021/12/08 17:58:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:58:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 17:58:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 17:58:22 Rebalance progress: 12.5
2021/12/08 17:58:27 Rebalance progress: 12.5
2021/12/08 17:58:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:58:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 17:58:37 Rebalance progress: 12.5
2021/12/08 17:58:42 Rebalance progress: 12.5
2021/12/08 17:58:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:58:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 17:58:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 17:58:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:58:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:59:18 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:59:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 17:59:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:59:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:59:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:59:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:59:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:59:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6105
2021/12/08 17:59:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:59:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 17:59:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 17:59:41 eventingSetup.go:46: No eventing-con process running
2021/12/08 17:59:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 17:59:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:59:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:59:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 17:59:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 17:59:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 17:59:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.12s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/08 18:00:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:00:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:00:05 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:00:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 18:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:00:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:00:23 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:00:23 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/08 18:00:23 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:00:23 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/08 18:00:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:00:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:37 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:00:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:00:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:00:38 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:00:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/08 18:00:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 18:00:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:00:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:00:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:00:59 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/08 18:01:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:01:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:01:14 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:01:14 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:01:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 18:01:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:01:20 Rebalance progress: 12.5
2021/12/08 18:01:25 Rebalance progress: 12.5
2021/12/08 18:01:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:01:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 18:01:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:01:35 Rebalance progress: 12.5
2021/12/08 18:01:40 Rebalance progress: 12.5
2021/12/08 18:01:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:01:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:01:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/08 18:01:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:01:51 Rebalance progress: 12.5
2021/12/08 18:01:56 Rebalance progress: 12.5
2021/12/08 18:02:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:02:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 18:02:06 Rebalance progress: 12.5
2021/12/08 18:02:11 Rebalance progress: 12.5
2021/12/08 18:02:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:02:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 18:02:21 Rebalance progress: 12.5
2021/12/08 18:02:26 Rebalance progress: 12.5
2021/12/08 18:02:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:02:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/08 18:02:36 Rebalance progress: 12.5
2021/12/08 18:02:41 Rebalance progress: 12.5
2021/12/08 18:02:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:02:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:02:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:02:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:02:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:03:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:03:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:03:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:03:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:03:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:03:16 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:03:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 18:03:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:03:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:03:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:03:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:03:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:03:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12205
2021/12/08 18:03:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:03:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:03:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:03:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:03:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:03:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:04:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:04:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:04:07 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/08 18:04:07 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:04:07 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/08 18:04:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:04:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:04:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:04:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:04:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:04:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (265.67s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/08 18:04:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:04:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:04:30 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:04:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 18:04:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:04:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:04:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:04:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:04:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:04:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:04:49 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:04:49 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/08 18:04:49 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:04:49 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/08 18:04:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:04:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:04:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:04:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:05:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:05:02 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:05:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:05:03 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:05:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/08 18:05:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 18:05:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:05:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:05:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:05:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/08 18:05:30 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:05:35 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:05:40 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/08 18:05:40 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:05:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 18:05:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:05:45 Rebalance progress: 12.5
2021/12/08 18:05:50 Rebalance progress: 12.5
2021/12/08 18:05:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:05:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:05:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 18:05:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 18:06:00 Rebalance progress: 12.5
2021/12/08 18:06:05 Rebalance progress: 12.5
2021/12/08 18:06:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:06:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:06:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:06:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 18:06:16 Rebalance progress: 12.5
2021/12/08 18:06:21 Rebalance progress: 12.5
2021/12/08 18:06:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:06:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:06:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:06:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:06:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:06:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:06:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 18:06:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:07:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:07:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7589
2021/12/08 18:07:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:07:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:07:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:07:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7547
2021/12/08 18:07:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7547
2021/12/08 18:07:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:07:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:07:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/08 18:07:56 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/08 18:07:56 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:07:56 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/08 18:07:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:08:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:08:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:08:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:08:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:08:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (229.94s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/08 18:08:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:08:20 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:08:20 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:08:20 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 18:08:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:08:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:08:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:08:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:08:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:08:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:08:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:08:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:08:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 18:08:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 18:08:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:08:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:09:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:09:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:09:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 18:09:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:09:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:09:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:09:36 Rebalance progress: 12.5
2021/12/08 18:09:41 Rebalance progress: 12.5
2021/12/08 18:09:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:09:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 18:09:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 18:09:51 Rebalance progress: 12.5
2021/12/08 18:09:56 Rebalance progress: 12.5
2021/12/08 18:10:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:10:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 18:10:06 Rebalance progress: 12.5
2021/12/08 18:10:11 Rebalance progress: 12.5
2021/12/08 18:10:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:10:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:10:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:10:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:10:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:10:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:10:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:10:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:10:42 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:10:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 18:10:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:10:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:10:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9149
2021/12/08 18:10:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9149
2021/12/08 18:10:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:11:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:11:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:11:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4380
2021/12/08 18:11:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4380
2021/12/08 18:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.25s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/08 18:11:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:11:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:11:25 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:11:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/08 18:11:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:11:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:11:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:11:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:11:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:11:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:11:44 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:11:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 18:11:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/08 18:11:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:12:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:12:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:12:05 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:12:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 18:12:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:12:10 Rebalance progress: 12.5
2021/12/08 18:12:15 Rebalance progress: 12.5
2021/12/08 18:12:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:12:20 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:12:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:12:36 Rebalance progress: 12.5
2021/12/08 18:12:41 Rebalance progress: 12.5
2021/12/08 18:12:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:12:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:12:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/08 18:12:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/08 18:12:51 Rebalance progress: 12.5
2021/12/08 18:12:56 Rebalance progress: 12.5
2021/12/08 18:13:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:13:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:13:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:13:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/08 18:13:06 Rebalance progress: 12.5
2021/12/08 18:13:11 Rebalance progress: 12.5
2021/12/08 18:13:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:13:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:13:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:13:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:13:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:13:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:13:46 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:13:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 18:13:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:13:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8598
2021/12/08 18:13:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:14:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:14:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:14:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 315
2021/12/08 18:14:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 315
2021/12/08 18:14:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.52s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/08 18:14:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/08 18:14:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/08 18:14:25 Rebalance progress: 100
2021/12/08 18:14:25 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:14:55 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/08 18:14:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/08 18:15:01 Removing node(s): [] from the cluster
2021/12/08 18:15:01 Starting bucket ops to source bucket
2021/12/08 18:15:07 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/08 18:15:12 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:15:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/08 18:15:17 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:15:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:15:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/08 18:15:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/08 18:15:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:15:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:15:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:15:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:15:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/08 18:15:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/08 18:15:42 eventingSetup.go:46: No eventing-con process running
2021/12/08 18:15:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/08 18:15:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:15:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:15:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/08 18:15:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1507
2021/12/08 18:15:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/08 18:16:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/08 18:16:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/08 18:16:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 858
2021/12/08 18:16:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 858
2021/12/08 18:16:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.56s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	11972.404s

Rebalance tests took 3 hours, 19 minutes and 35 seconds.

Testing Succeeded


Version: versions-08.12.2021-10.39.cfg
Build Log: make-08.12.2021-10.39.log
Server Log: logs-08.12.2021-10.39.tar.gz

Finished