Building

Started building at 2021/12/01 11:38:27
Calculating base
Updating mirror
Basing run on 7.1.0-1814 6c2474ff9
Updating tree for run 01.12.2021-11.38
gocb is at 635fa6c, changes since last good build: 
 635fa6c Prepare v2.2.3 release
 480b951 Bump gocbcore version
 e2f9f32 GOCBC-1074: Update threshold logger tag names
 0fceb9c Increase resilience of bucket flush test
 036928f Bump gocbcore version
 d174e08 Bump gocbcore version
 4b41226 Prepare v2.2.2 release
 0e9af9c Close results from setting up analytics query testing
 b6d7a57 GOCBC-1032: Add an internal function for verifying capability status
 bbee470 GOCBC-1070: Return bucket not found from flush when bucket does not exist
 5bd0533 Tweak the error message we look for for manager collection not found
 b29bec0 Update core tracer wrapper to match gocbcore interface
 3f96627 GOCBC-1024: Add partition field to QueryIndex
 be0cba9 Add extra nolint: unused tags
 d4bf8bf GOCBC-1010: Add collections support to search query options
 ef5d355 Add named parameter query tests
 2ade4bb Extend expiry conversion test window allowance
 546f3e9 Bump version to v2.2.1
 e7fccc9 GOCBC-1054: Allow blank paths to be used with subdoc remove
 c59f1d8 GOCBC-1040: Include error causes when doing serialization
 33094ff GOCBC-1061: Assign role origins via index rather than append on get
 8ee25cf GOCBC-1047: Add MaxExpiry to collections from GetAllScopes
 d98a3f6 GOCBC-1048: Update collection manager error message parsing
 7b5d7a3 Add test feature flags for fine tuning collections testing
 73e8d9c GOCBC-1017: Update collection manager rest API calls
 6b401f5 GOCBC-1055: Add doc flags as an internal mutate in option
 1b71002 Enable out of order responses by default
 0ab256b GOCBC-1052: Filter by provided namespace for get all design docs
 6123f2a Increase *query test indexing deadline to 60 seconds
 0d6f7b0 GOCBC-1031: Add SubdocDocFlagCreateAsDeleted flag as internal
 dd3ab11 Update user mgr tests to use replication_admin not security_admin
 ad350bc Remove modules-download-mode from .golangci.yml
 9e841df Bump version to v2.2.0
 e692eb8 Gardening: Update scope level (analytics) query to be volatile api
 e2bcf30 Update to latest gocbcore
 9b2060c GOCBC-854: Add support for user impersonation
 e29525f GOCBC-1023: Add interfaces for accessing raw bytes from http services
 fed377a GOCBC-934: Add support for bucket level durability
 d0e869d Populate bucket MaxExpiry from MaxTTL
 75a1e63 GOCBC-1014: Mark geopolygon query as api stability committed
 ac59a0a GOCBC-1015: Mark flexindex as api stability committed
 6c3c74a GOCBC-1026: Mark ephemeral bucket eviction policies as api committed
 f8b7f33 GOCBC-967: Increase timeout of first durability op in durability tests
 b8929c8 GOCBC-972: Add support for disabling scoring
 67f9a2f GOCBC-963: Deprecate GetResult.Expiry
 9285de0 GOCBC-948: Refactor document expiry duration
 3368e21 GOCBC-869: Deprecate BucketSettings MaxTTL and add MaxExpiry
 625accf GOCBC-1021: Route bucket level http requests to agents not agent group
 9b54ac3 GOCBC-1022: Route view queries to agents not agent group
 2fe9d0b Bump version to v2.1.8
 1057287 Update tests to not expect retries from Unlock
 d67e05d Bump gocbcore version
 37b0923 GOCBC-1005: Add document id to key value error
 d53d2a4 GOCBC-937: Add support for GeoPolygon search queries.
 171294e Bump version to v2.1.7
 2898a81 Bump gocbcore version to latest
 68da463 GOCBC-1001: Add missing fields to search facets results
 3745ce1 GOCBC-944: Add uncommitted support for collections analytics
 48721f3 GOCBC-994:  Convert subdoc nil values to json null
 c64613e Check for replica readyness on test startup
 fb060ef Update kv op manager tests to create deadline at comparison time
 cc0f0a4 GOCBC-996: Remove `[]` from id for `CollectionMap.At` and `Exist`
 3fe96bb GOCBC-971: Add uncommitted support for collections to user management
 77d100b GOCBC-942: Add uncommitted support for collections queries
 2fe8c14 GOCBC-990: Update enhanced durability adaptive algorithm
 c9c6097 GOCBC-977: Alter signature of GetPendingMutations
 efb249b Increase resilience of TestMaxNumberOfCollectionInScope
 582acb1 GOCBC-938: Add support for flex indexes in query options
 9cd266e Bump version to v2.1.6
 33c0f62 Add query index test feature flag so index crud tests can be disabled
 61882ae Bump gocbcore version to get latest changes
 ca37170 Bump gocbcore version
 c364430 GOCBC-980: Update get projections with expiry result parsing
 5c7b802 GOCBC-976: Allow WithExpiry to work with custom transcoders
 0993c98 GOCBC-979: Add ExpiryTime to GetResult
 42c87f8 GOCBC-969: Only try to parse query metrics if present
 94cf884 SDKQE-2028:Collections and Scopes Max Limit Tests
 f31151b GOCBC-970: Run query, search, view tests against default collection
libcouchbase is at f6822fc, changes since last good build: 
 f6822fcc CCBC-1519: pass extra privilege with KV "on-behalf-of"
 38cc0708 Update release meta for 3.2.4
 d54c7953 CCBC-1518: prepare for git:// protocol deprecation
 3f29165e CCBC-1522: Check DnsQuery result type
 16463b31 CCBC-1521: fix bootstrap process when client cert is used
 b17efe63 CCBC-1509: record more info in TLS log callback
 4a12295e SDKQE-2448: Integration test for LCB HTTP bootstrap
 a0cc39cf Remove short timeout in change password test to prevent jenkins failures
 eb4e471e CCBC-1432: rename rate-limit error codes for consistency
 644194e5 CCBC-1432: check for rate-limit errors during bootstrap
 1b607a69 CCBC-1432: support for rate limiting error codes
 256ec277 CCBC-1514: do not translate unknown error with "item-only" attribute
 e6a5bd8f CCBC-1515: replace sstream with string append()
 0cddb2cd report SASL mechanisms, supported by the server when PLAIN is being rejected
 a6407406 report if PKCS5_PBKDF2_HMAC was available during the build
 b3294247 Update release meta for 3.2.3
 2f75ba87 CCBC-1446: clarify structure of scope qualifier for analytics
 c1a04a6f Decrease mock test time
 b8772261 SDKQE-2328: OpenTelemetry metrics testing
 5a5df389 SDKQE-2324: OpenTelemetry tracing testing
 d57e4ccd CCBC-1484: fix tracing tags in accordance to RFC
indexing is at a00b35f, changes since last good build: 
 a00b35f1 Merge remote-tracking branch 'couchbase/unstable' into HEAD
 9dcb9a41 MB-43886 : Add basic clusterInfoClientLite
 19232c79 MB-48896:Implement fail-fast mechanism in metadata_provider for free tier limits
 ffdf1d62 MB-46068 Force disk snapshot with maintFlushTs during merge
 5cda891c Merge remote-tracking branch 'couchbase/unstable' into HEAD
 2eff3732 MB-49743 : Dont omit replica id when 0
 7627b880 MB-48229: Fix race between build index command and it's token processing
 df4a2737 Merge remote-tracking branch 'couchbase/unstable' into HEAD
 b5c2ed8c MB-43886: Fetch data on Hash Change in PoolChangeNotification
 6831b39b MB-49356 Part 1 (7.1.0 1695): Autofailover: IsSafe() more logging
 fa06f68b MB-49501 Part 3 (7.1.0 1695): Autofailover: Make CPU throttling safer
 e8c2cc3c MB-49501 Part 2 (7.1.0 1695): Autofailover: Make CPU throttling safer
 e3673def Merge remote-tracking branch 'couchbase/unstable' into HEAD
 3b29b923 MB-49700: Account for compressed records in resident_percent
 0cbc503a MB-46251 Handle errors in shutdownVBuckets code path
 2a2c4cdc MB-44775: All array index show wrong item indexed count in UI
 e9350384 MB-49694 (7.1.0 1695): Smart Batching: Additional logging
 4b9d513e Merge remote-tracking branch 'couchbase/unstable' into HEAD
 adac97c0 MB-49591 Check seq. order violation for snapshot messages
 d8ba487e MB-49662 Part 1 (7.1.0 1650): Autofailover: Make CPU throttling safer
kv_engine is at dd35c1f, changes since last good build: 
 dd35c1f11 [Magma] Check status of getCollectionsManifestUidDoc
 821bed179 MB-49626: getDroppedCollectionStats -> getDroppedCollectionItemCount
 3e1ba42c4 MB-49626: Test different state in FailCompactKVStoreCall
 96411e5bd MB-49803: Check ctx before using it in setMakeCompactionCallback
 0d011d3a4 Revert "MB-45563: get_collection|scope_id: input in value only"
 e975b77b2 MB-48659: Remove redundant code
 c70ce0859 MB-49767: Add extra info to exception message in PDM
 3735366fd MB-47300: Add CouchKVStore::logExistingVBucketFiles
 aa34d9676 MB-43460: Fix collection accounting bug when col dropped but not flushed
 f0f4444d0 Clang-Tidy: vbucket.(h/cc)
 d84cd789a MB-49626: Move docCount update into CompactKVStore()
 6d65851b3 TSAN: Ensure we don't try and access freed ThreadGate
 71b4ca052 Add new method KVBucketTest::replaceMagmaKVStore() to simplify tests
 a390d1e8c MB-48637: Change PersistManifestTask from AuxIO to Writer
 eeb315140 Revert "MB-43360: Fix collection accounting bug when col dropped but not flushed"
 0b9f2131b MB-43360: Fix collection accounting bug when col dropped but not flushed
 9293cb5cf KVBucketTest: Add flushVBucket()
 27c9a9106 MB-49607: Fix doc size accounting when flushing in MagmaKVStore
 919fada27 MB-47300: Remove CouchKVstore::openSpecificDB
 a1983d430 Modify ADM::State::updateHighCompletedSeqno to throw base type
 0381a94c7 MB-49453: Don't generate a new snapshot when ending on a new checkpoint
 f3ae9691e KVBucketTest: Avoid unnecessary exception throwing
 dbbbfb424 Merge "Merge branch 'cheshire-cat'"
 20fd017ab MB-49526: Update MagmaCompactionCB to set it's magma::Status
 2586145f1 MB-47604: Shard skipGetWithHeaderChecksForRollback per vBucket
 81e2f5980 MB-49594: Checkpoint isn't friend of CheckpointCursor
 c67a7b712 MB-47604: Print manifest rather than uid when incorrect after flush
 97f221811 MB-49526: Modify MagmaKVStore MagmaCompactionCB & compactionCallBack
 f92ac63c6 MB-47604: Move cache slot logic from getLock to separate function
 7d4c0152a Merge branch 'cheshire-cat'
 da44f93a0 MB-49634: Propagate CompactKVStore failure up to caller
 d79b30546 MB-49742: Don't update droppedCollection if CompactKVStore fails
 1d2c671d0 MB-47604: Implicit compaction support for NexusKVStore
 9c94fd759 Merge "Merge branch 'cheshire-cat'"
 a7172f67c MB-49594: CheckpointManager isn't friend of CheckpointCursor
 b2401bc1b Merge branch 'cheshire-cat'
 d80763edd Merge branch 'cheshire-cat'
 87b80d95c Merge "Merge branch 'cheshire-cat'"
 59e58f6a8 MB-49207: Fix compaction expiry bg fetch race
 c1be8da0b MB-49207: Fix no item meta only fetch race
 69b511331 MB-47687: Add test functions to MockMagmaKVStore
 a059b6017 Merge branch 'cheshire-cat'
 da9d758e3 Merge "Merge branch 'cheshire-cat'"
 ba8ebe1fb MB-49027: Fix for bg fetch race meta only success path
 9c1e76e66 MB-49207: Fix BgFetcher race restoring old items for full fetches
 d90ecfdf5 MB-49594: Reset cursor distance in CM::resetCursors
 250a4bd57 Merge branch 'cheshire-cat'
 35e0f6116 MB-49594: Move CheckpointCursor code to its own file
 31a3baa21 MB-49594: Simplify cursor-repositioning logic in CM::addNewCheckpoint
 b24093b32 Merge branch 'mad-hatter' into cheshire-cat
 89bb095e9 Merge branch 'mad-hatter' into cheshire-cat
 fbdf0d1e1 MB-49022: Merge branch 'mad-hatter' into cheshire-cat
 954523f51 Merge branch 'mad-hatter' into cheshire-cat
 57e005f0f MB-48923: Merge commit '93350e0d6' into cheshire-cat
 153910b2d MB-43238: Merge branch '6.6.3' into mad-hatter
 cf9166197 MB-43238: Merge branch '6.6.3' into mad-hatter
 121f613b7 MB-43238: [BP] Don't expire committed items if prepare exists
 e1e0fac94 MB-43238: [BP] Take a FindUpdateResult in VB::processExpiredItem
 42e953cf8 Merge branch 'mad-hatter' into cheshire-cat
 3aeccbc77 MB-49022: Merge branch '6.6.3' into mad-hatter
 15286301b MB-49022: Always reset the SV committed state on expiry
 c049b8f9f [BP] MB-48398: Don't retry a stream-request with an incompatible UUID
 93350e0d6 MB-48923: Avoid underflow of on_disk_prepare_bytes during compaction
 aa44e4504 MB-48713: [BP] Set correct revSeqno for sync write add for Ephemeral
ns_server is at 9756174, changes since last good build: 
 97561740c MB-49568: Add limits global descriptive key in error message
 3b6b85a7a Merge "Merge remote-tracking branch 'couchbase/cheshire-cat'"
 6ba5cb678 Add otp_node field in event logs added by Services
 0125e8991 Add otp_node field to all event logs
 3b2d51282 Merge remote-tracking branch 'couchbase/cheshire-cat'
 477a70dd3 MB-48767 Add global default test
 e67b1067a MB-48767 Add incomplete settings test
 d2e8e3be0 MB-48767 Require complete time periods
 9f21c6a94 MB-48767 Keep autocompaction settings when not specified
 005e060c2 MB-49735: Merge branch 'mad-hatter' into cheshire-cat
 e346f8969 MB-49750: Prevent monitoring of sockets as processes
 942918405 MB-49735 [backport] repair chronicle and ns_config after...
 3f5120395 MB-49794 Correct JSON-fication errors in auto_failover_warning event log
 d5e37e6d9 MB-49745 Disallow loading sample buckets on mixed versions
 40c349c00 MB-49791 Exclude failover_reason in event log if it is empty
 59c34120b MB-49778 Fix a typo in hard_failover_initated event log
 ac784350e MB-47387 Increase ns_server->memcached timeout
 582121bd5 MB-48446: disable navigation to the same page twice
 d03855231 MB-49086 Add inline spinner directive
 f630d9d4b MB-49310: add currentnav class if 'app.admin.groups' present in state
 3cc4040fe MB-49184: permissions: hide General Settings Analytics Replicas field
 4616dbbc0  MB-49184: use right permissions to protect Views section
 7c32b8e12 MB-49184: permissions: disable sample buckets sumbit btn
 f19535dad MB-49184: permissions: disable sample buckets checkboxes
 c53493958 MB-49184: completely hide footer if user has no permissions
 0d1b8beb4 MB-49184: cluster.settings!read is required by Sample Bucket page
 3353db4fe MB-49184: cluster.settings!read is required by Alerts page
 367b98914 MB-49184: hasWritePermissions should be defined before use
 c049de072 MB-49184: disable 'Enable Bloom Filter' if no cluster.settings.write
 ce201afe8 Turn unknown types and function warnings into errors.
 892d34be6 Avoid duplication in do-dialyzer.
 0584804f4 MB-49451 Removed unused variables
 67d04a2c2 MB-49451 Add nodeServicesUri to pools/default response
 744d3e73d MB-48063: Do quorum read on manifest for collection update
 8555ed747 MB-49681 Disable 'load sample buckets' button in mixed version mode
 43c483fa6 User Roles: auto-submit keyspace-selector values when step is 'ok'
 233305c75 MB-49680: Ignore files that start with '.'
 a9b70a942 MB-49142 Allow Eventing Manage Scope Functions to see stats
 bd9b7c845 MB-48738: depend on pluggable ui js.stamp only if project src dir exists
 6b7d74564 Deal with "unknown function" warnings in calls to loggers.
 1536237ed MB-49129: Don't display 'Show Current Collection' for idle status
 ddd500bc9 MB-49578: Collect Info: open Stop Collection modal from form page.
 f5376025c Drop unused include.
 f86588873 Fix indentation in ale_transform.
 61c428128 Fix tls_config_timer type specification.
 e43b8d325 Stop using unexported ssl:reason/0 type.
 bd7e220c9 Correctly refer to the set() type.
 5e08b84d9 Add missing dependencies to the PLT file.
 17dc0211d Include couchdb apps into the PLT file.
 ae94452e3 Fix unreachable clause in the to chronicle:remove_peer/2.
 31d13ad1f Remove dead code from do-dialyzer.cmake.
 1c95a7de2 Update the license header for scripts/escript-wrapper.
 c740dc743 MB-42727 Deal with monitor() vs {whereis,register}_name() races.
 d5aabf066 Demonitor registered processes when leader changes.
 75431d75f Implement leader_registry:unregister_name/1.
 261b2debb Add a basic test for leader_registry.
 2e1ec68a3 Merge leader_registry and leader_registry_server.
 2f8f45555 Stop supporting switching back to global-based orchestration.
 7f150f9ee MB-48199 supporting CSS for QWB
 ab0947fce MB-49697: Missing 'rest' in alternate ports update
 c1b9d7db0 MB-48637 Increase allowable timeout for ensureManifest
 e5beadbd1 MB-49663 Remove nonsensical title - sample buckets
 bb3dd84bd Support QWB prompt to make primary index
 426fe742a MB-49069: add check for cluster.settings.metrics permissions
 ad9a7aa1e MB-49414: check bucket any permission if bucket isn't provided
 fcf1b061a MB-49414: add importmap for mnDetailStats
 28c806df4 MB-49414: call define method on ace in snippets
couchstore is at 05c4320, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 6061aba, changes since last good build: 
 6061aba8b MB-49593: disable action=failover test cases
 68051a19b Fix old x509 jobs
 a616e1638 add 6.5.2, 6.6.3, 7.0.0 and 7.0.1 to release builds
 7a714f20a CBQE-7348: Add support for system event log checking in backup service
 aa4b01644 CBQE-7328: Merging sys_event_log lib changes from TAF
 40e7ae360 CBQE-7208: 28/n Initial upgrade tests
 e1afb32c3 dummy suite setup/teardown stable_topo fts
 9a9fdbdde Remove unnecessary bucket delete retries
 7f296230e CBQE-7349: Add retry to TLS enforcement checking
 ab9ec0afd Eventing: Multiple root CA and encrypted private key tests
Switching eventing to unstable
eventing is at e4e62e2, changes since last good build: 
 e4e62e27 MB-49418: Check cached entry for app state
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 52f6f102] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 8f5df5ce] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 718f39ea] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166486 at refs/changes/86/166486/3
Total 7 (delta 2), reused 2 (delta 2)
[unstable 261755f5] MB-49148: Grant SystemXattrWrite privilege to on_behalf_of_ user
 Author: AnkitPrabhu 
 Date: Sat Nov 27 15:22:26 2021 +0530
 2 files changed, 31 insertions(+), 13 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/01 11:53:57
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/01 11:55:42 Started functional
2021/12/01 11:55:51 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 11:55:58 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/01 11:55:58 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 11:56:03 Removing node(s): [] from the cluster
2021/12/01 11:56:08 Rebalance progress: 2.67650462962963
2021/12/01 11:56:13 Rebalance progress: 7.269965277777778
2021/12/01 11:56:18 Rebalance progress: 10.47092013888889
2021/12/01 11:56:23 Rebalance progress: 15.0462962962963
2021/12/01 11:56:28 Rebalance progress: 18.51851851851852
2021/12/01 11:56:33 Rebalance progress: 22.85879629629629
2021/12/01 11:56:38 Rebalance progress: 27.39800347222222
2021/12/01 11:56:43 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/01 11:56:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:57:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:57:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:57:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:57:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:58:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:58:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:58:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:58:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:59:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:59:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:59:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 11:59:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:00:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:00:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:00:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:00:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:01:04 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:01:19 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:01:34 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:01:49 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/01 12:02:04 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:02:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:02:10 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 12:02:15 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 12:02:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 12:02:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/01 12:02:25 App: TestAdvancedGetOps got deployed
2021/12/01 12:02:25 Starting bucket ops to source bucket
2021/12/01 12:02:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:02:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:02:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:02:35 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:02:35 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:02:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:02:40 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 12:02:45 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 12:02:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 12:02:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/01 12:02:55 App: TestAdvancedGetOps got un-deployed
2021/12/01 12:02:55 No eventing-con process running
2021/12/01 12:02:55 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/01 12:02:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:03:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:03:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:03:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:03:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:03:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:03:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (395.19s)
=== RUN   TestAdvancedInsertOps
2021/12/01 12:03:24 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:03:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:03:30 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:03:35 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:03:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:03:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:03:45 App: TestAdvancedInsertOps got deployed
2021/12/01 12:03:45 Starting bucket ops to source bucket
2021/12/01 12:03:51 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 12:03:51 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 12:03:51 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:03:51 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:03:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:03:56 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:04:01 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:04:06 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:04:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:04:11 App: TestAdvancedInsertOps got un-deployed
2021/12/01 12:04:11 No eventing-con process running
2021/12/01 12:04:11 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/01 12:04:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:04:23 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:04:28 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:04:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:04:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:04:40 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:04:45 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:04:45 Testing insert operation on source bucket
2021/12/01 12:04:45 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:04:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:04:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:04:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:05:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:05:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/01 12:05:06 App: TestAdvancedInsertOps got deployed
2021/12/01 12:05:06 Starting bucket ops to source bucket
2021/12/01 12:05:12 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 12:05:12 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 12:05:12 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:05:12 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:05:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:05:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:05:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:05:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:05:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/01 12:05:32 App: TestAdvancedInsertOps got un-deployed
2021/12/01 12:05:32 No eventing-con process running
2021/12/01 12:05:32 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/01 12:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:05:44 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/01 12:05:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:05:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:05:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.57s)
=== RUN   TestAdvancedUpsertOps
2021/12/01 12:05:55 Starting bucket ops to source bucket
2021/12/01 12:05:56 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:05:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:06:02 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:06:07 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:06:12 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:06:17 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:06:17 App: TestAdvancedUpsertOps got deployed
2021/12/01 12:06:17 Starting bucket ops to source bucket
2021/12/01 12:06:23 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:06:28 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:06:33 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:06:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:06:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:06:33 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:06:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:06:38 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:06:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:06:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:06:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:06:53 App: TestAdvancedUpsertOps got un-deployed
2021/12/01 12:06:53 No eventing-con process running
2021/12/01 12:06:53 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/01 12:06:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:07:05 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:07:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:07:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:07:12 Testing upsert operation on source bucket
2021/12/01 12:07:12 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:07:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:07:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:07:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:07:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:07:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/01 12:07:33 App: TestAdvancedUpsertOps got deployed
2021/12/01 12:07:33 Starting bucket ops to source bucket
2021/12/01 12:07:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:07:34 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:07:34 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:07:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:07:39 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:07:44 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:07:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:07:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/01 12:07:54 App: TestAdvancedUpsertOps got un-deployed
2021/12/01 12:07:54 No eventing-con process running
2021/12/01 12:07:54 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/01 12:07:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:08:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.99s)
=== RUN   TestAdvancedReplaceOps
2021/12/01 12:08:08 Starting bucket ops to source bucket
2021/12/01 12:08:09 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:08:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:08:15 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:08:20 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:08:25 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:08:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:08:30 App: TestAdvancedReplaceOps got deployed
2021/12/01 12:08:30 Starting bucket ops to source bucket
2021/12/01 12:08:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:08:41 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:08:47 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/01 12:08:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:08:47 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:08:47 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:08:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:08:52 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:08:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:09:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:09:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:09:07 App: TestAdvancedReplaceOps got un-deployed
2021/12/01 12:09:07 No eventing-con process running
2021/12/01 12:09:07 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/01 12:09:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:09:18 Waiting for dst bucket item count to get to: 0 curr count: 817
2021/12/01 12:09:23 Waiting for dst bucket item count to get to: 0 curr count: 305
2021/12/01 12:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:09:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:09:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:09:30 Testing upsert operation on source bucket
2021/12/01 12:09:30 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:09:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:09:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:09:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:09:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:09:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/01 12:09:51 App: TestAdvancedReplaceOps got deployed
2021/12/01 12:09:51 Starting bucket ops to source bucket
2021/12/01 12:09:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:09:52 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:09:52 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:09:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:09:57 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:10:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:10:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:10:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/01 12:10:13 App: TestAdvancedReplaceOps got un-deployed
2021/12/01 12:10:13 No eventing-con process running
2021/12/01 12:10:13 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/01 12:10:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:10:24 Waiting for dst bucket item count to get to: 0 curr count: 329
2021/12/01 12:10:29 Waiting for dst bucket item count to get to: 0 curr count: 158
2021/12/01 12:10:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:10:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:10:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (147.56s)
=== RUN   TestAdvancedDeleteOps
2021/12/01 12:10:36 Starting bucket ops to source bucket
2021/12/01 12:10:36 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:10:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:10:42 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:10:47 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:10:52 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:10:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:10:57 App: TestAdvancedDeleteOps got deployed
2021/12/01 12:10:57 Starting bucket ops to source bucket
2021/12/01 12:11:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:11:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:11:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:11:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:11:13 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:11:13 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:11:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:11:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:11:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:11:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:11:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:11:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/01 12:11:33 No eventing-con process running
2021/12/01 12:11:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/01 12:11:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:11:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:11:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:11:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:11:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:11:51 Testing delete operation on source bucket
2021/12/01 12:11:51 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:11:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/01 12:11:57 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:12:02 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:12:07 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:12:12 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/01 12:12:12 App: TestAdvancedDeleteOps got deployed
2021/12/01 12:12:12 Starting bucket ops to source bucket
2021/12/01 12:12:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:12:13 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:12:13 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:12:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:12:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:12:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:12:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:12:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/01 12:12:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/01 12:12:33 No eventing-con process running
2021/12/01 12:12:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/01 12:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:12:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:12:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.69s)
=== RUN   TestEnoentAdvancedGet
2021/12/01 12:12:47 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:12:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:12:53 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:12:58 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:13:03 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:13:08 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:13:08 App: TestEnoentAdvancedGet got deployed
2021/12/01 12:13:08 Starting bucket ops to source bucket
2021/12/01 12:13:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:13:18 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:13:23 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:13:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:13:23 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:13:23 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:13:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:13:28 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:13:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:13:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:13:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:13:43 App: TestEnoentAdvancedGet got un-deployed
2021/12/01 12:13:43 No eventing-con process running
2021/12/01 12:13:43 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/01 12:13:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:13:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:13:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:14:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:14:07 Testing get enoent operation on source bucket
2021/12/01 12:14:07 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:14:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:14:13 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:14:18 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:14:23 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:14:28 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/01 12:14:28 App: TestEnoentAdvancedGet got deployed
2021/12/01 12:14:28 Starting bucket ops to source bucket
2021/12/01 12:14:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 12:14:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 12:14:43 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 12:14:43 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 12:14:43 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:14:43 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:14:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:14:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:14:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:14:58 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:15:03 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/01 12:15:03 App: TestEnoentAdvancedGet got un-deployed
2021/12/01 12:15:03 No eventing-con process running
2021/12/01 12:15:03 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/01 12:15:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:15:15 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 12:15:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:15:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:15:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.19s)
=== RUN   TestEnoentAdvancedReplace
2021/12/01 12:15:22 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:15:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:15:28 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:15:33 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:15:38 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:15:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:15:43 App: TestEnoentAdvancedReplace got deployed
2021/12/01 12:15:43 Starting bucket ops to source bucket
2021/12/01 12:15:48 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:15:53 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:15:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:15:53 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:15:53 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:15:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:15:58 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:16:03 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:16:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:16:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:16:13 App: TestEnoentAdvancedReplace got un-deployed
2021/12/01 12:16:13 No eventing-con process running
2021/12/01 12:16:13 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/01 12:16:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:16:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:16:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:16:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:16:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:16:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:16:37 Testing get enoent operation on source bucket
2021/12/01 12:16:37 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:16:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:16:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:16:48 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:16:53 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:16:58 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/01 12:16:58 App: TestEnoentAdvancedReplace got deployed
2021/12/01 12:16:58 Starting bucket ops to source bucket
2021/12/01 12:17:03 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 12:17:08 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 12:17:08 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 12:17:08 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:17:08 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:17:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:17:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:17:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:17:23 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:17:28 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/01 12:17:28 App: TestEnoentAdvancedReplace got un-deployed
2021/12/01 12:17:28 No eventing-con process running
2021/12/01 12:17:28 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/01 12:17:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:17:40 Waiting for dst bucket item count to get to: 0 curr count: 139
2021/12/01 12:17:45 Waiting for dst bucket item count to get to: 0 curr count: 139
2021/12/01 12:17:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:17:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:17:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.09s)
=== RUN   TestEnoentAdvancedDelete
2021/12/01 12:17:52 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:17:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 12:17:58 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:18:03 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:18:08 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:18:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:18:13 App: TestEnoentAdvancedDelete got deployed
2021/12/01 12:18:13 Starting bucket ops to source bucket
2021/12/01 12:18:18 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:18:23 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:18:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:18:23 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:18:23 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:18:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:18:28 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:18:33 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:18:38 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:18:43 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:18:43 App: TestEnoentAdvancedDelete got un-deployed
2021/12/01 12:18:43 No eventing-con process running
2021/12/01 12:18:44 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/01 12:18:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:18:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:18:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:19:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:19:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:19:07 Testing get enoent operation on source bucket
2021/12/01 12:19:07 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:19:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 12:19:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:19:18 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:19:23 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:19:28 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/01 12:19:28 App: TestEnoentAdvancedDelete got deployed
2021/12/01 12:19:28 Starting bucket ops to source bucket
2021/12/01 12:19:33 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/01 12:19:38 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/01 12:19:38 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/01 12:19:38 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:19:38 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:19:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:19:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:19:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:19:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:19:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/01 12:19:59 App: TestEnoentAdvancedDelete got un-deployed
2021/12/01 12:19:59 No eventing-con process running
2021/12/01 12:19:59 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/01 12:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:20:10 Waiting for dst bucket item count to get to: 0 curr count: 148
2021/12/01 12:20:15 Waiting for dst bucket item count to get to: 0 curr count: 148
2021/12/01 12:20:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.05s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/01 12:20:22 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:20:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/01 12:20:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:20:33 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:20:38 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:20:43 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:20:43 App: TestAdvancedInsertKeyExist got deployed
2021/12/01 12:20:43 Starting bucket ops to source bucket
2021/12/01 12:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:20:43 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:20:43 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:20:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:20:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:20:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:20:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:21:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:21:04 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/01 12:21:04 No eventing-con process running
2021/12/01 12:21:04 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/01 12:21:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:21:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:21:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:21:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:21:18 Pumping document in the destination bucket
2021/12/01 12:21:18 Starting bucket ops to source bucket
2021/12/01 12:21:18 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:21:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/01 12:21:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:21:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:21:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:21:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/01 12:21:39 App: TestAdvancedInsertKeyExist got deployed
2021/12/01 12:21:39 Starting bucket ops to source bucket
2021/12/01 12:21:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:21:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:21:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:21:49 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:21:49 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:21:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:21:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:21:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:22:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:22:09 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/01 12:22:09 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/01 12:22:09 No eventing-con process running
2021/12/01 12:22:09 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/01 12:22:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:22:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:22:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:22:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:22:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:22:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.88s)
=== RUN   TestExipryGet
2021/12/01 12:22:33 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:22:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:22:39 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:22:44 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:22:49 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:22:54 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:22:54 App: TestExipryGet got deployed
2021/12/01 12:22:54 Starting bucket ops to source bucket
2021/12/01 12:23:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:23:44 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:23:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:23:44 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:23:44 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:23:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:23:49 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:23:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:24:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:24:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:24:05 App: TestExipryGet got un-deployed
2021/12/01 12:24:05 No eventing-con process running
2021/12/01 12:24:05 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/01 12:24:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:24:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:24:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:24:19 Pumping document in the destination bucket for expiry get
2021/12/01 12:24:19 Starting bucket ops to source bucket
2021/12/01 12:24:19 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:24:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:24:25 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:24:30 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:24:35 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:24:40 Waiting for app: TestExipryGet to get deployed
2021/12/01 12:24:40 App: TestExipryGet got deployed
2021/12/01 12:24:40 Starting bucket ops to source bucket
2021/12/01 12:25:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:25:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:25:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:25:30 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:25:30 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:25:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:25:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:25:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:25:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:25:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/01 12:25:50 App: TestExipryGet got un-deployed
2021/12/01 12:25:50 No eventing-con process running
2021/12/01 12:25:50 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/01 12:25:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:26:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:26:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:26:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:26:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:26:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.95s)
=== RUN   TestExipryInsert
2021/12/01 12:26:14 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:26:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/01 12:26:20 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:26:25 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:26:30 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:26:35 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:26:35 App: TestExipryInsert got deployed
2021/12/01 12:26:35 Starting bucket ops to source bucket
2021/12/01 12:27:20 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/01 12:27:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:27:20 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:27:20 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:27:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:27:25 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:27:30 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:27:35 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:27:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:27:40 App: TestExipryInsert got un-deployed
2021/12/01 12:27:40 No eventing-con process running
2021/12/01 12:27:40 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/01 12:27:42 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:27:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:27:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:27:55 Testing on the destination bucket for expiry insert
2021/12/01 12:27:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:27:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/01 12:28:01 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:28:06 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:28:11 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:28:16 Waiting for app: TestExipryInsert to get deployed
2021/12/01 12:28:16 App: TestExipryInsert got deployed
2021/12/01 12:28:16 Starting bucket ops to source bucket
2021/12/01 12:29:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:29:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:29:01 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:29:01 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:29:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:29:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:29:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:29:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:29:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/01 12:29:21 App: TestExipryInsert got un-deployed
2021/12/01 12:29:21 No eventing-con process running
2021/12/01 12:29:21 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/01 12:29:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:29:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:29:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:29:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.72s)
=== RUN   TestExipryReplace
2021/12/01 12:29:45 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:29:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:29:51 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:29:56 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:30:01 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:30:06 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:30:06 App: TestExipryReplace got deployed
2021/12/01 12:30:06 Starting bucket ops to source bucket
2021/12/01 12:30:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:30:51 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:30:51 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:30:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:30:56 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:31:01 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:31:06 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:31:11 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:31:11 App: TestExipryReplace got un-deployed
2021/12/01 12:31:11 No eventing-con process running
2021/12/01 12:31:11 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/01 12:31:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:31:20 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:31:25 Testing on the destination bucket for expiry insert
2021/12/01 12:31:25 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:31:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:31:31 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:31:36 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:31:41 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:31:46 Waiting for app: TestExipryReplace to get deployed
2021/12/01 12:31:46 App: TestExipryReplace got deployed
2021/12/01 12:31:46 Starting bucket ops to source bucket
2021/12/01 12:32:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:32:32 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:32:32 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:32:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:32:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:32:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:32:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:32:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/01 12:32:52 App: TestExipryReplace got un-deployed
2021/12/01 12:32:52 No eventing-con process running
2021/12/01 12:32:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/01 12:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:33:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:33:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:33:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.77s)
=== RUN   TestExipryUpsert
2021/12/01 12:33:15 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:33:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/01 12:33:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:33:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:33:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:33:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:33:37 App: TestExipryUpsert got deployed
2021/12/01 12:33:37 Starting bucket ops to source bucket
2021/12/01 12:34:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:34:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:34:22 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:34:22 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:34:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:34:27 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:34:32 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:34:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:34:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:34:42 App: TestExipryUpsert got un-deployed
2021/12/01 12:34:42 No eventing-con process running
2021/12/01 12:34:42 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/01 12:34:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:34:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:34:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:34:56 Testing on the destination bucket for expiry insert
2021/12/01 12:34:56 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:34:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/01 12:35:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:35:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:35:12 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:35:17 Waiting for app: TestExipryUpsert to get deployed
2021/12/01 12:35:17 App: TestExipryUpsert got deployed
2021/12/01 12:35:17 Starting bucket ops to source bucket
2021/12/01 12:36:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:12 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:36:17 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:36:17 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:36:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:36:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:36:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:36:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:36:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/01 12:36:38 App: TestExipryUpsert got un-deployed
2021/12/01 12:36:38 No eventing-con process running
2021/12/01 12:36:38 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/01 12:36:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:36:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:36:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:37:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (225.81s)
=== RUN   TestCasReplace
2021/12/01 12:37:01 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:37:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:37:07 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:37:12 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:37:17 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:37:22 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:37:22 App: TestCasReplace got deployed
2021/12/01 12:37:22 Starting bucket ops to source bucket
2021/12/01 12:37:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:37:23 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:37:23 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:37:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:37:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:37:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:37:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:37:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:37:43 App: TestCasReplace got un-deployed
2021/12/01 12:37:43 No eventing-con process running
2021/12/01 12:37:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/01 12:37:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:37:55 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/01 12:37:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:37:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:38:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:38:01 Testing on the destination bucket for upsert with cas
2021/12/01 12:38:01 Starting bucket ops to source bucket
2021/12/01 12:38:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:38:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/01 12:38:08 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:38:13 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:38:18 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:38:23 Waiting for app: TestCasReplace to get deployed
2021/12/01 12:38:23 App: TestCasReplace got deployed
2021/12/01 12:38:23 Starting bucket ops to source bucket
2021/12/01 12:38:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:38:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:38:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:38:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:38:33 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:38:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:38:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:38:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:38:48 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:38:53 Waiting for app: TestCasReplace to get un-deployed
2021/12/01 12:38:53 App: TestCasReplace got un-deployed
2021/12/01 12:38:53 No eventing-con process running
2021/12/01 12:38:53 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/01 12:38:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:39:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:39:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.33s)
=== RUN   TestCasDelete
2021/12/01 12:39:12 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:39:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 12:39:18 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:39:23 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:39:28 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:39:33 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:39:33 App: TestCasDelete got deployed
2021/12/01 12:39:33 Starting bucket ops to source bucket
2021/12/01 12:39:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:39:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:39:33 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:39:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:39:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:39:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:39:48 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:39:53 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:39:53 App: TestCasDelete got un-deployed
2021/12/01 12:39:53 No eventing-con process running
2021/12/01 12:39:53 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/01 12:39:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:40:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:40:07 Testing on the destination bucket for delete with cas
2021/12/01 12:40:07 Starting bucket ops to source bucket
2021/12/01 12:40:07 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:40:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 12:40:13 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:40:19 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:40:24 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:40:29 Waiting for app: TestCasDelete to get deployed
2021/12/01 12:40:29 App: TestCasDelete got deployed
2021/12/01 12:40:29 Starting bucket ops to source bucket
2021/12/01 12:40:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:40:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:40:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:40:39 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:40:39 http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 12:40:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:40:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:40:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:40:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:40:59 Waiting for app: TestCasDelete to get un-deployed
2021/12/01 12:40:59 App: TestCasDelete got un-deployed
2021/12/01 12:40:59 No eventing-con process running
2021/12/01 12:40:59 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/01 12:41:01 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:41:11 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/01 12:41:16 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/01 12:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:41:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.83s)
=== RUN   TestCountersIncrement
2021/12/01 12:41:22 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 12:41:29 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 12:41:29 Removing node(s): [] from the cluster
2021/12/01 12:41:34 Rebalance progress: 100
2021/12/01 12:41:34 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:41:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/01 12:41:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:41:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:41:50 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:41:55 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:41:55 App: TestCountersIncrement got deployed
2021/12/01 12:41:55 Starting bucket ops to source bucket
2021/12/01 12:42:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 12:42:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/01 12:42:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 12:42:06 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:42:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:42:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:42:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:42:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:42:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:42:27 App: TestCountersIncrement got un-deployed
2021/12/01 12:42:27 No eventing-con process running
2021/12/01 12:42:27 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/01 12:42:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:42:38 Waiting for dst bucket item count to get to: 0 curr count: 822
2021/12/01 12:42:43 Waiting for dst bucket item count to get to: 0 curr count: 309
2021/12/01 12:42:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:42:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:42:55 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:42:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:42:55 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:42:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/01 12:43:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:43:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:43:11 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:43:16 Waiting for app: TestCountersIncrement to get deployed
2021/12/01 12:43:16 App: TestCountersIncrement got deployed
2021/12/01 12:43:16 Starting bucket ops to source bucket
2021/12/01 12:43:23 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 12:43:28 Waiting for dst bucket item count to get to: 2048 curr count: 1670
2021/12/01 12:43:33 Waiting for dst bucket item count to get to: 2048 curr count: 1670
2021/12/01 12:43:33 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 12:43:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:43:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:43:38 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:43:43 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:43:48 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:43:53 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/01 12:43:53 App: TestCountersIncrement got un-deployed
2021/12/01 12:43:53 No eventing-con process running
2021/12/01 12:43:53 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/01 12:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:44:05 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/01 12:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:44:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:44:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 12:44:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:44:16 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/01 12:44:21 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.12s)
=== RUN   TestCountersDecrement
2021/12/01 12:44:21 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 12:44:22 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 12:44:22 Removing node(s): [] from the cluster
2021/12/01 12:44:27 Rebalance progress: 100
2021/12/01 12:44:27 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:44:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/01 12:44:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:44:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:44:43 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:44:48 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:44:48 App: TestCountersDecrement got deployed
2021/12/01 12:44:48 Starting bucket ops to source bucket
2021/12/01 12:44:55 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/01 12:45:00 Waiting for dst bucket item count to get to: 1024 curr count: 406
2021/12/01 12:45:05 Waiting for dst bucket item count to get to: 1024 curr count: 406
2021/12/01 12:45:05 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/01 12:45:05 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:45:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:45:10 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:45:15 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:45:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:45:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:45:25 App: TestCountersDecrement got un-deployed
2021/12/01 12:45:25 No eventing-con process running
2021/12/01 12:45:25 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/01 12:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:45:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/01 12:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:45:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:45:48 Waiting for dst bucket item count to get to: 0 curr count: 838
2021/12/01 12:45:53 Waiting for dst bucket item count to get to: 0 curr count: 326
2021/12/01 12:45:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:45:53 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:45:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/01 12:45:59 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:46:05 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:46:10 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:46:15 Waiting for app: TestCountersDecrement to get deployed
2021/12/01 12:46:15 App: TestCountersDecrement got deployed
2021/12/01 12:46:15 Starting bucket ops to source bucket
2021/12/01 12:46:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/01 12:46:26 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/01 12:46:26 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/01 12:46:26 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:46:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:46:31 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:46:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:46:41 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:46:46 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/01 12:46:46 App: TestCountersDecrement got un-deployed
2021/12/01 12:46:46 No eventing-con process running
2021/12/01 12:46:46 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/01 12:46:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:46:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/01 12:46:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:47:04 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/01 12:47:09 Rebalance progress: 100
--- PASS: TestCountersDecrement (167.97s)
=== RUN   TestCacheGet
2021/12/01 12:47:09 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:47:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:47:16 Waiting for app: TestCacheGet to get deployed
2021/12/01 12:47:21 Waiting for app: TestCacheGet to get deployed
2021/12/01 12:47:26 Waiting for app: TestCacheGet to get deployed
2021/12/01 12:47:31 Waiting for app: TestCacheGet to get deployed
2021/12/01 12:47:31 App: TestCacheGet got deployed
2021/12/01 12:47:31 Starting bucket ops to source bucket
2021/12/01 12:47:36 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/01 12:47:41 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/01 12:47:46 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/01 12:47:51 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/01 12:47:56 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/01 12:48:01 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/01 12:48:01 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/01 12:48:01 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:48:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:48:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 12:48:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 12:48:16 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 12:48:21 Waiting for app: TestCacheGet to get un-deployed
2021/12/01 12:48:21 App: TestCacheGet got un-deployed
2021/12/01 12:48:21 No eventing-con process running
2021/12/01 12:48:21 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/01 12:48:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:48:33 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/01 12:48:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/01 12:48:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:48:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:48:50 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/01 12:48:55 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/01 12:48:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.29s)
=== RUN   TestCacheRYOW
2021/12/01 12:48:55 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:48:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/01 12:49:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 12:49:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 12:49:11 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 12:49:16 Waiting for app: TestCacheRYOW to get deployed
2021/12/01 12:49:16 App: TestCacheRYOW got deployed
2021/12/01 12:49:16 Starting bucket ops to source bucket
2021/12/01 12:49:21 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/01 12:49:21 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/01 12:49:21 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:49:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:49:26 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 12:49:31 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 12:49:36 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 12:49:41 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/01 12:49:41 App: TestCacheRYOW got un-deployed
2021/12/01 12:49:41 No eventing-con process running
2021/12/01 12:49:41 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/01 12:49:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:49:53 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:49:58 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/01 12:49:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:50:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:50:10 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/01 12:50:15 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/01 12:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.23s)
=== RUN   TestCacheExpiry
2021/12/01 12:50:15 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:50:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/01 12:50:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 12:50:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 12:50:31 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 12:50:36 Waiting for app: TestCacheExpiry to get deployed
2021/12/01 12:50:36 App: TestCacheExpiry got deployed
2021/12/01 12:50:36 Starting bucket ops to source bucket
2021/12/01 12:50:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:50:46 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 12:50:51 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 12:50:56 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 12:51:01 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 12:51:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:51:01 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:51:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:51:07 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 12:51:12 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 12:51:17 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 12:51:22 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/01 12:51:22 App: TestCacheExpiry got un-deployed
2021/12/01 12:51:22 No eventing-con process running
2021/12/01 12:51:22 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/01 12:51:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:51:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:51:38 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/01 12:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:51:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:51:50 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/01 12:51:55 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/01 12:51:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.24s)
=== RUN   TestCacheExpiryCustom
2021/12/01 12:51:55 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:51:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/01 12:52:01 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 12:52:06 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 12:52:11 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 12:52:16 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/01 12:52:16 App: TestCacheExpiryCustom got deployed
2021/12/01 12:52:16 Starting bucket ops to source bucket
2021/12/01 12:52:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:52:27 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 12:52:32 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/01 12:52:37 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 12:52:42 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/01 12:52:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:52:42 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:52:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:52:47 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 12:52:52 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 12:52:57 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 12:53:02 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/01 12:53:02 App: TestCacheExpiryCustom got un-deployed
2021/12/01 12:53:02 No eventing-con process running
2021/12/01 12:53:02 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/01 12:53:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:53:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:53:19 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/01 12:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:53:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:53:31 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/01 12:53:36 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/01 12:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.38s)
=== RUN   TestCacheOverflowCustom
2021/12/01 12:53:36 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:53:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/01 12:53:42 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 12:53:47 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 12:53:52 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 12:53:57 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/01 12:53:57 App: TestCacheOverflowCustom got deployed
2021/12/01 12:53:57 Starting bucket ops to source bucket
2021/12/01 12:54:02 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/01 12:54:07 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/01 12:54:12 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/01 12:54:17 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/01 12:54:22 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/01 12:54:22 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/01 12:54:22 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:54:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:54:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 12:54:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 12:54:37 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 12:54:42 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/01 12:54:42 App: TestCacheOverflowCustom got un-deployed
2021/12/01 12:54:42 No eventing-con process running
2021/12/01 12:54:42 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/01 12:54:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:54:54 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/01 12:54:59 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/01 12:54:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:55:01 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:55:11 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 12:55:16 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 12:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.28s)
=== RUN   TestStrictMode
2021/12/01 12:55:16 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:55:17 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/01 12:55:17 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:55:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/01 12:55:23 Waiting for app: TestDataTypes to get deployed
2021/12/01 12:55:28 Waiting for app: TestDataTypes to get deployed
2021/12/01 12:55:33 Waiting for app: TestDataTypes to get deployed
2021/12/01 12:55:38 Waiting for app: TestDataTypes to get deployed
2021/12/01 12:55:38 App: TestDataTypes got deployed
2021/12/01 12:55:38 Starting bucket ops to source bucket
2021/12/01 12:55:43 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/01 12:55:48 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/01 12:55:53 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/01 12:55:53 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/01 12:55:53 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:55:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:55:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 12:56:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 12:56:08 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 12:56:13 Waiting for app: TestDataTypes to get un-deployed
2021/12/01 12:56:13 App: TestDataTypes got un-deployed
2021/12/01 12:56:13 No eventing-con process running
2021/12/01 12:56:13 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/01 12:56:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:56:25 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 12:56:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:56:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:56:37 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/01 12:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.14s)
=== RUN   TestEnoentGet
2021/12/01 12:56:37 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:56:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:56:43 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:56:48 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:56:53 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:56:58 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:56:58 App: TestEnoentGet got deployed
2021/12/01 12:56:58 Starting bucket ops to source bucket
2021/12/01 12:57:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:57:08 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:57:08 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:57:08 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:57:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:57:14 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:57:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:57:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:57:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:57:29 App: TestEnoentGet got un-deployed
2021/12/01 12:57:29 No eventing-con process running
2021/12/01 12:57:29 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/01 12:57:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:57:40 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/01 12:57:45 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/01 12:57:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:57:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:57:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 12:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:57:57 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/01 12:58:03 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:58:08 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:58:14 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:58:19 Waiting for app: TestEnoentGet to get deployed
2021/12/01 12:58:19 App: TestEnoentGet got deployed
2021/12/01 12:58:19 Starting bucket ops to source bucket
2021/12/01 12:58:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:58:29 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:58:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:58:29 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:58:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:58:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:58:39 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:58:44 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:58:49 Waiting for app: TestEnoentGet to get un-deployed
2021/12/01 12:58:49 App: TestEnoentGet got un-deployed
2021/12/01 12:58:49 No eventing-con process running
2021/12/01 12:58:49 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/01 12:58:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:59:01 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/01 12:59:06 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/01 12:59:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 12:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 12:59:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 12:59:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.42s)
=== RUN   TestEnoentDelete
2021/12/01 12:59:18 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 12:59:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 12:59:24 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 12:59:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 12:59:34 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 12:59:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 12:59:39 App: TestEnoentDelete got deployed
2021/12/01 12:59:39 Starting bucket ops to source bucket
2021/12/01 12:59:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 12:59:49 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 12:59:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 12:59:49 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 12:59:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 12:59:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 12:59:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:00:04 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:00:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:00:09 App: TestEnoentDelete got un-deployed
2021/12/01 13:00:09 No eventing-con process running
2021/12/01 13:00:09 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/01 13:00:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:00:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 13:00:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 13:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:00:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:00:38 Waiting for dst bucket item count to get to: 0 curr count: 56
2021/12/01 13:00:43 Waiting for dst bucket item count to get to: 0 curr count: 6
2021/12/01 13:00:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:00:43 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:00:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 13:00:49 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 13:00:54 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 13:00:59 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 13:01:04 Waiting for app: TestEnoentDelete to get deployed
2021/12/01 13:01:04 App: TestEnoentDelete got deployed
2021/12/01 13:01:04 Starting bucket ops to source bucket
2021/12/01 13:01:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 13:01:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/01 13:01:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 13:01:14 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:01:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:01:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:01:24 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:01:29 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:01:34 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/01 13:01:34 App: TestEnoentDelete got un-deployed
2021/12/01 13:01:34 No eventing-con process running
2021/12/01 13:01:34 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/01 13:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:01:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 13:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:01:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:01:58 Waiting for dst bucket item count to get to: 0 curr count: 66
2021/12/01 13:02:03 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/01 13:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (165.49s)
=== RUN   TestError
2021/12/01 13:02:13 Waiting for app: TestError to get un-deployed
2021/12/01 13:02:13 App: TestError got un-deployed
2021/12/01 13:02:13 No eventing-con process running
2021/12/01 13:02:13 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/01 13:02:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:02:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:02:22 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:02:27 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:02:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/01 13:02:33 Waiting for app: TestError to get deployed
2021/12/01 13:02:38 Waiting for app: TestError to get deployed
2021/12/01 13:02:43 Waiting for app: TestError to get deployed
2021/12/01 13:02:48 Waiting for app: TestError to get deployed
2021/12/01 13:02:48 App: TestError got deployed
2021/12/01 13:02:48 Starting bucket ops to source bucket
2021/12/01 13:02:53 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/01 13:02:58 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/01 13:03:03 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/01 13:03:03 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/01 13:03:03 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:03:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:03:08 Waiting for app: TestError to get un-deployed
2021/12/01 13:03:13 Waiting for app: TestError to get un-deployed
2021/12/01 13:03:18 Waiting for app: TestError to get un-deployed
2021/12/01 13:03:23 Waiting for app: TestError to get un-deployed
2021/12/01 13:03:23 App: TestError got un-deployed
2021/12/01 13:03:23 No eventing-con process running
2021/12/01 13:03:23 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/01 13:03:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:03:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 13:03:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:03:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:03:47 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/01 13:03:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.74s)
=== RUN   TestCRLF
2021/12/01 13:03:57 Waiting for app: TestCRLF to get un-deployed
2021/12/01 13:03:57 App: TestCRLF got un-deployed
2021/12/01 13:03:57 No eventing-con process running
2021/12/01 13:03:57 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/01 13:03:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:04:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:04:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:04:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:04:11 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:04:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/01 13:04:17 Waiting for app: TestCRLF to get deployed
2021/12/01 13:04:22 Waiting for app: TestCRLF to get deployed
2021/12/01 13:04:27 Waiting for app: TestCRLF to get deployed
2021/12/01 13:04:32 Waiting for app: TestCRLF to get deployed
2021/12/01 13:04:32 App: TestCRLF got deployed
2021/12/01 13:04:32 Starting bucket ops to source bucket
2021/12/01 13:04:37 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 13:04:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 13:04:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 13:04:42 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:04:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:04:47 Waiting for app: TestCRLF to get un-deployed
2021/12/01 13:04:52 Waiting for app: TestCRLF to get un-deployed
2021/12/01 13:04:57 Waiting for app: TestCRLF to get un-deployed
2021/12/01 13:05:02 Waiting for app: TestCRLF to get un-deployed
2021/12/01 13:05:02 App: TestCRLF got un-deployed
2021/12/01 13:05:02 No eventing-con process running
2021/12/01 13:05:02 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/01 13:05:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:05:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 13:05:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 13:05:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:05:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:05:31 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/01 13:05:36 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/01 13:05:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (109.02s)
=== RUN   TestImportExport
2021/12/01 13:05:46 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:05:46 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 13:05:46 No eventing-con process running
2021/12/01 13:05:46 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/01 13:05:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:05:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:05:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:05:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:06:04 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:06:04 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 13:06:04 No eventing-con process running
2021/12/01 13:06:04 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/01 13:06:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:06:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:06:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:06:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:06:18 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:06:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 13:06:19 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:06:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 13:06:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 13:06:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 13:06:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 13:06:41 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/01 13:06:41 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/01 13:06:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 13:06:51 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 13:06:56 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/01 13:06:56 App: TestImportExport_bucket_op_on_update got deployed
2021/12/01 13:06:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:07:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:07:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:07:11 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:07:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:07:16 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 13:07:16 No eventing-con process running
2021/12/01 13:07:16 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/01 13:07:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:07:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:07:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:07:31 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:07:36 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:07:36 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 13:07:36 No eventing-con process running
2021/12/01 13:07:36 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/01 13:07:53 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/01 13:07:53 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/01 13:07:53 No eventing-con process running
2021/12/01 13:07:53 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/01 13:07:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:08:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:08:12 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/01 13:08:12 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/01 13:08:12 No eventing-con process running
2021/12/01 13:08:12 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/01 13:08:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:08:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.98s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/01 13:08:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:08:36 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:08:36 No eventing-con process running
2021/12/01 13:08:36 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/01 13:08:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:08:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:08:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:08:49 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:08:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:08:51 Starting bucket ops to source bucket
2021/12/01 13:08:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:09:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:09:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:09:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:09:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:09:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 13:09:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 13:09:19 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:09:19 Undeploying app: bucket_op_on_update
2021/12/01 13:09:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:09:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:09:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:09:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:09:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:09:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:09:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:09:39 No eventing-con process running
2021/12/01 13:09:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:09:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:10:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:10:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:10:24 Starting bucket ops to source bucket
2021/12/01 13:10:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:10:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:10:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:10:47 Waiting for dst bucket item count to get to: 5000 curr count: 1802
2021/12/01 13:10:52 Waiting for dst bucket item count to get to: 5000 curr count: 4302
2021/12/01 13:10:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:10:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 13:10:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 13:10:57 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:10:57 Undeploying app: bucket_op_on_update
2021/12/01 13:10:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:11:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:11:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:11:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:11:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:11:17 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:11:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:11:17 No eventing-con process running
2021/12/01 13:11:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:11:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:12:02 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:12:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:12:03 Starting bucket ops to source bucket
2021/12/01 13:12:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:12:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:12:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:12:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:12:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:12:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 13:12:31 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 13:12:31 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:12:31 Undeploying app: bucket_op_on_update
2021/12/01 13:12:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:12:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:12:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:12:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:12:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:12:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:12:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:12:51 No eventing-con process running
2021/12/01 13:12:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:13:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:13:35 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:13:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:13:36 Starting bucket ops to source bucket
2021/12/01 13:13:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:13:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:13:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:13:59 Waiting for dst bucket item count to get to: 5000 curr count: 1535
2021/12/01 13:14:04 Waiting for dst bucket item count to get to: 5000 curr count: 1535
2021/12/01 13:14:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:14:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 13:14:09 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 13:14:09 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:14:09 Undeploying app: bucket_op_on_update
2021/12/01 13:14:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:14:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:14:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:14:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:14:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:14:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:14:29 No eventing-con process running
2021/12/01 13:14:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:14:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:15:14 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:15:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:15:15 Starting bucket ops to source bucket
2021/12/01 13:15:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:15:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:15:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:15:38 Waiting for dst bucket item count to get to: 5000 curr count: 751
2021/12/01 13:15:43 Waiting for dst bucket item count to get to: 5000 curr count: 3251
2021/12/01 13:15:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:15:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/01 13:15:48 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/01 13:15:48 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:15:48 Undeploying app: bucket_op_on_update
2021/12/01 13:15:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:15:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:15:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:16:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:16:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/01 13:16:08 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/01 13:16:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:16:08 No eventing-con process running
2021/12/01 13:16:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:16:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:16:52 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (506.25s)
=== RUN   TestOnUpdateN1QLOp
2021/12/01 13:17:02 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 13:17:02 App: TestOnUpdateN1QLOp got un-deployed
2021/12/01 13:17:02 No eventing-con process running
2021/12/01 13:17:02 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/01 13:17:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:17:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:17:16 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:17:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/01 13:17:17 Starting bucket ops to source bucket
2021/12/01 13:17:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:17:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:17:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:17:40 Waiting for dst bucket item count to get to: 5000 curr count: 238
2021/12/01 13:17:45 Waiting for dst bucket item count to get to: 5000 curr count: 238
2021/12/01 13:17:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:17:45 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:17:50 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 13:17:55 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 13:18:00 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 13:18:05 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/01 13:18:05 App: TestOnUpdateN1QLOp got un-deployed
2021/12/01 13:18:05 No eventing-con process running
2021/12/01 13:18:05 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/01 13:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:18:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:18:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:18:29 Waiting for dst bucket item count to get to: 0 curr count: 4645
2021/12/01 13:18:34 Waiting for dst bucket item count to get to: 0 curr count: 2102
2021/12/01 13:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.76s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/01 13:18:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 13:18:44 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/01 13:18:44 No eventing-con process running
2021/12/01 13:18:44 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/01 13:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:18:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:18:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:18:57 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:18:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:18:59 Starting bucket ops to source bucket
2021/12/01 13:19:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:19:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:19:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:19:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:19:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 13:19:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:19:27 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:19:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:19:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 13:19:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 13:19:42 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 13:19:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/01 13:19:47 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/01 13:19:47 No eventing-con process running
2021/12/01 13:19:47 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/01 13:19:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:19:59 Waiting for dst bucket item count to get to: 0 curr count: 4557
2021/12/01 13:20:04 Waiting for dst bucket item count to get to: 0 curr count: 2057
2021/12/01 13:20:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:20:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.84s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/01 13:20:26 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 13:20:26 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 13:20:26 No eventing-con process running
2021/12/01 13:20:26 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/01 13:20:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:20:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:20:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:20:39 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:20:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:20:40 Starting bucket ops to source bucket
2021/12/01 13:20:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:20:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:20:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:21:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:21:03 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:21:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:21:08 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 13:21:13 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 13:21:18 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 13:21:24 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 13:21:24 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 13:21:24 No eventing-con process running
2021/12/01 13:21:24 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/01 13:21:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:21:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:21:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (91.60s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/01 13:21:57 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 13:21:57 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/01 13:21:57 No eventing-con process running
2021/12/01 13:21:57 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/01 13:21:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:22:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:22:11 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:22:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/01 13:22:12 Starting bucket ops to source bucket
2021/12/01 13:22:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:22:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:22:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:22:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:22:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:22:35 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:22:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:22:40 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 13:22:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 13:22:50 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 13:22:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/01 13:22:55 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/01 13:22:55 No eventing-con process running
2021/12/01 13:22:55 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/01 13:22:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:23:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:23:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:23:19 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:23:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 13:23:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.78s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/01 13:23:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 13:23:34 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/01 13:23:34 No eventing-con process running
2021/12/01 13:23:34 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/01 13:23:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:23:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:23:43 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:23:48 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:23:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/01 13:23:49 Starting bucket ops to source bucket
2021/12/01 13:23:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:24:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:24:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:24:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:24:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 13:24:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:24:17 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:24:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:24:22 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 13:24:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 13:24:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 13:24:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/01 13:24:38 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/01 13:24:38 No eventing-con process running
2021/12/01 13:24:38 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/01 13:24:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:24:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 13:24:54 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 13:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:24:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:25:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.34s)
=== RUN   TestOnDeleteBucketOp
2021/12/01 13:25:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 13:25:16 App: TestOnDeleteBucketOp got un-deployed
2021/12/01 13:25:16 No eventing-con process running
2021/12/01 13:25:16 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/01 13:25:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:25:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:25:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:25:30 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:25:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 13:25:31 Starting bucket ops to source bucket
2021/12/01 13:25:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:25:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:25:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:25:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:25:52 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:25:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:25:57 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 13:26:02 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 13:26:07 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 13:26:12 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/01 13:26:12 App: TestOnDeleteBucketOp got un-deployed
2021/12/01 13:26:12 No eventing-con process running
2021/12/01 13:26:12 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/01 13:26:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:26:32 Waiting for dst bucket item count to get to: 0 curr count: 4859
2021/12/01 13:26:37 Waiting for dst bucket item count to get to: 0 curr count: 4859
2021/12/01 13:26:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.20s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/01 13:26:47 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 13:26:47 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/01 13:26:47 No eventing-con process running
2021/12/01 13:26:47 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/01 13:26:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:26:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:26:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:27:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:27:00 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:27:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/01 13:27:01 Starting bucket ops to source bucket
2021/12/01 13:27:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:27:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:27:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:27:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:27:22 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:27:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:27:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 13:27:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 13:27:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 13:27:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/01 13:27:43 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/01 13:27:43 No eventing-con process running
2021/12/01 13:27:43 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/01 13:27:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:27:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:27:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:28:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:28:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.33s)
=== RUN   TestTimerBucketOp
2021/12/01 13:28:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 13:28:17 App: TestTimerBucketOp got un-deployed
2021/12/01 13:28:17 No eventing-con process running
2021/12/01 13:28:17 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/01 13:28:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:28:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:28:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:28:30 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:28:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 13:28:32 Starting bucket ops to source bucket
2021/12/01 13:28:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:28:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:29:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:29:10 Waiting for dst bucket item count to get to: 5000 curr count: 1353
2021/12/01 13:29:15 Waiting for dst bucket item count to get to: 5000 curr count: 1353
2021/12/01 13:29:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:29:15 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:29:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:29:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 13:29:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 13:29:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 13:29:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/01 13:29:35 App: TestTimerBucketOp got un-deployed
2021/12/01 13:29:35 No eventing-con process running
2021/12/01 13:29:35 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/01 13:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:29:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:29:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:29:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:29:58 Waiting for dst bucket item count to get to: 0 curr count: 4278
2021/12/01 13:30:03 Waiting for dst bucket item count to get to: 0 curr count: 1778
2021/12/01 13:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.55s)
=== RUN   TestTimerInPastBucketOp
2021/12/01 13:30:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 13:30:13 App: TestTimerInPastBucketOp got un-deployed
2021/12/01 13:30:13 No eventing-con process running
2021/12/01 13:30:13 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/01 13:30:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:30:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:30:22 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:30:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:30:27 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:30:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 13:30:28 Starting bucket ops to source bucket
2021/12/01 13:30:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:30:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:30:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:30:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:30:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:31:01 Waiting for dst bucket item count to get to: 5000 curr count: 24
2021/12/01 13:31:06 Waiting for dst bucket item count to get to: 5000 curr count: 2524
2021/12/01 13:31:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:31:06 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:31:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:31:11 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 13:31:16 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 13:31:21 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 13:31:26 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/01 13:31:26 App: TestTimerInPastBucketOp got un-deployed
2021/12/01 13:31:26 No eventing-con process running
2021/12/01 13:31:26 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/01 13:31:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:31:38 Waiting for dst bucket item count to get to: 0 curr count: 3601
2021/12/01 13:31:43 Waiting for dst bucket item count to get to: 0 curr count: 1101
2021/12/01 13:31:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:31:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:31:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:31:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.59s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/01 13:32:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:32:05 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:32:05 No eventing-con process running
2021/12/01 13:32:05 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/01 13:32:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:32:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:32:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:32:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:32:19 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:32:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 13:32:20 Starting bucket ops to source bucket
2021/12/01 13:32:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:32:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:32:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:32:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:32:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:32:53 Waiting for dst bucket item count to get to: 5000 curr count: 160
2021/12/01 13:32:58 Waiting for dst bucket item count to get to: 5000 curr count: 1409
2021/12/01 13:33:03 Waiting for dst bucket item count to get to: 5000 curr count: 3749
2021/12/01 13:33:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:33:08 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 13:33:08 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 13:33:08 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:33:08 Undeploying app: bucket_op_with_timer
2021/12/01 13:33:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:33:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:33:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:33:23 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:33:28 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:33:28 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:33:28 No eventing-con process running
2021/12/01 13:33:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:34:12 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:34:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 13:34:13 Starting bucket ops to source bucket
2021/12/01 13:34:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:34:56 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 13:34:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:35:01 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 13:35:01 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 13:35:01 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:35:01 Undeploying app: bucket_op_with_timer
2021/12/01 13:35:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:35:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:35:12 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:35:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:35:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:35:22 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:35:22 No eventing-con process running
2021/12/01 13:35:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:35:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:36:06 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:36:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 13:36:07 Starting bucket ops to source bucket
2021/12/01 13:36:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:36:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:36:50 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 13:36:50 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 13:36:50 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:36:50 Undeploying app: bucket_op_with_timer
2021/12/01 13:36:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:36:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:37:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:37:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:37:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:37:10 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:37:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:37:10 No eventing-con process running
2021/12/01 13:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:37:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:37:54 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:37:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 13:37:55 Starting bucket ops to source bucket
2021/12/01 13:38:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:38:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:38:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:38:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:38:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:38:28 Waiting for dst bucket item count to get to: 5000 curr count: 589
2021/12/01 13:38:33 Waiting for dst bucket item count to get to: 5000 curr count: 589
2021/12/01 13:38:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:38:38 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 13:38:38 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 13:38:38 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:38:38 Undeploying app: bucket_op_with_timer
2021/12/01 13:38:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:38:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:38:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:38:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:38:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:38:58 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:38:58 No eventing-con process running
2021/12/01 13:39:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:39:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:39:43 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:39:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 13:39:44 Starting bucket ops to source bucket
2021/12/01 13:39:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:39:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:40:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 13:40:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:40:32 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/01 13:40:32 App: TestDeployUndeployLoopTimer got deployed
2021/12/01 13:40:32 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:40:32 Undeploying app: bucket_op_with_timer
2021/12/01 13:40:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:40:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:40:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:40:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:40:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/01 13:40:52 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/01 13:40:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:40:52 No eventing-con process running
2021/12/01 13:40:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:41:01 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:41:36 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (581.01s)
=== RUN   TestMultipleHandlers
2021/12/01 13:41:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 13:41:46 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/01 13:41:46 No eventing-con process running
2021/12/01 13:41:46 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/01 13:41:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:41:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:41:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:42:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:42:05 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 13:42:05 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/01 13:42:05 No eventing-con process running
2021/12/01 13:42:05 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/01 13:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:42:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:42:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:42:18 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 13:42:19 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:42:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/01 13:42:26 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 13:42:31 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 13:42:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 13:42:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/01 13:42:41 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/01 13:42:46 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 13:42:51 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 13:42:56 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/01 13:42:56 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/01 13:42:56 Starting bucket ops to source bucket
2021/12/01 13:43:05 Waiting for dst bucket item count to get to: 5000 curr count: 782
2021/12/01 13:43:10 Waiting for dst bucket item count to get to: 5000 curr count: 2679
2021/12/01 13:43:15 Waiting for dst bucket item count to get to: 5000 curr count: 4397
2021/12/01 13:43:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:43:15 Starting bucket ops to source bucket
2021/12/01 13:43:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:43:30 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 13:43:30 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:43:30 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:43:30 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:43:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:43:35 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 13:43:40 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 13:43:45 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 13:43:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/01 13:43:50 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/01 13:43:50 No eventing-con process running
2021/12/01 13:43:51 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/01 13:43:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:43:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:44:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:44:10 Waiting for dst bucket item count to get to: 0 curr count: 5004
2021/12/01 13:44:15 Waiting for dst bucket item count to get to: 0 curr count: 5004
2021/12/01 13:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:44:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:44:20 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 13:44:25 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 13:44:30 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 13:44:35 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/01 13:44:35 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/01 13:44:35 No eventing-con process running
2021/12/01 13:44:35 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/01 13:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:44:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:44:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.63s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/01 13:44:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 13:44:59 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/01 13:44:59 No eventing-con process running
2021/12/01 13:44:59 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/01 13:45:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:45:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:45:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:45:12 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:45:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:45:13 Starting bucket ops to source bucket
2021/12/01 13:45:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:45:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:45:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:45:36 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 13:45:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:45:36 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:45:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 13:45:41 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 13:45:41 Pausing app: bucket_op_on_update
2021/12/01 13:45:42 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:45:47 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 13:45:47 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 13:45:47 Resuming app: bucket_op_on_update
2021/12/01 13:45:48 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:45:48 Starting bucket ops to source bucket
2021/12/01 13:45:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:46:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:46:06 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/01 13:46:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 13:46:06 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:46:11 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 13:46:11 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 13:46:11 Pausing app: bucket_op_on_update
2021/12/01 13:46:11 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:46:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 13:46:16 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 13:46:16 Resuming app: bucket_op_on_update
2021/12/01 13:46:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:46:17 Starting bucket ops to source bucket
2021/12/01 13:46:25 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:46:30 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:46:35 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/01 13:46:35 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/01 13:46:35 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:46:40 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 13:46:40 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 13:46:40 Pausing app: bucket_op_on_update
2021/12/01 13:46:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:46:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 13:46:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 13:46:45 Resuming app: bucket_op_on_update
2021/12/01 13:46:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:46:46 Starting bucket ops to source bucket
2021/12/01 13:46:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:46:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:47:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:47:04 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 13:47:04 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:47:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 13:47:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 13:47:09 Pausing app: bucket_op_on_update
2021/12/01 13:47:09 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:47:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 13:47:15 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 13:47:15 Resuming app: bucket_op_on_update
2021/12/01 13:47:16 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:47:16 Starting bucket ops to source bucket
2021/12/01 13:47:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:47:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:47:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:47:34 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 13:47:34 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:47:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/01 13:47:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/01 13:47:39 Pausing app: bucket_op_on_update
2021/12/01 13:47:39 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:47:44 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/01 13:47:44 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/01 13:47:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:47:49 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 13:47:54 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 13:47:59 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 13:48:04 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/01 13:48:04 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/01 13:48:04 No eventing-con process running
2021/12/01 13:48:04 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/01 13:48:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:48:16 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 13:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:48:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:48:28 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/01 13:48:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (218.89s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/01 13:48:38 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 13:48:38 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/01 13:48:38 No eventing-con process running
2021/12/01 13:48:38 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/01 13:48:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:48:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:48:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:48:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:48:51 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:48:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 13:48:52 Starting bucket ops to source bucket
2021/12/01 13:49:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:49:30 Waiting for dst bucket item count to get to: 5000 curr count: 3366
2021/12/01 13:49:35 Waiting for dst bucket item count to get to: 5000 curr count: 3366
2021/12/01 13:49:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:49:35 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:49:40 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 13:49:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 13:49:40 Pausing app: bucket_op_with_timer
2021/12/01 13:49:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:49:45 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 13:49:45 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 13:49:45 Resuming app: bucket_op_with_timer
2021/12/01 13:49:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:49:47 Starting bucket ops to source bucket
2021/12/01 13:49:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:50:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:50:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:50:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:50:15 Waiting for dst bucket item count to get to: 10000 curr count: 5861
2021/12/01 13:50:20 Waiting for dst bucket item count to get to: 10000 curr count: 8361
2021/12/01 13:50:20 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 13:50:20 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:50:25 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 13:50:25 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 13:50:25 Pausing app: bucket_op_with_timer
2021/12/01 13:50:25 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:50:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 13:50:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 13:50:30 Resuming app: bucket_op_with_timer
2021/12/01 13:50:31 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:50:31 Starting bucket ops to source bucket
2021/12/01 13:50:39 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:50:44 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:50:49 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:50:54 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:50:59 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/01 13:51:04 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/01 13:51:04 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/01 13:51:04 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:51:09 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 13:51:09 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 13:51:09 Pausing app: bucket_op_with_timer
2021/12/01 13:51:09 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:51:14 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 13:51:14 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 13:51:14 Resuming app: bucket_op_with_timer
2021/12/01 13:51:15 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:51:15 Starting bucket ops to source bucket
2021/12/01 13:51:23 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:51:28 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:51:33 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:51:38 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:51:43 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/01 13:51:48 Waiting for dst bucket item count to get to: 20000 curr count: 15995
2021/12/01 13:51:53 Waiting for dst bucket item count to get to: 20000 curr count: 18495
2021/12/01 13:51:53 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 13:51:53 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:51:58 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 13:51:58 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 13:51:58 Pausing app: bucket_op_with_timer
2021/12/01 13:51:58 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:52:03 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 13:52:03 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 13:52:03 Resuming app: bucket_op_with_timer
2021/12/01 13:52:04 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:52:04 Starting bucket ops to source bucket
2021/12/01 13:52:12 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:52:17 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:52:22 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:52:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:52:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/01 13:52:37 Waiting for dst bucket item count to get to: 25000 curr count: 24565
2021/12/01 13:52:42 Waiting for dst bucket item count to get to: 25000 curr count: 24565
2021/12/01 13:52:42 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 13:52:42 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:52:47 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/01 13:52:48 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/01 13:52:48 Pausing app: bucket_op_with_timer
2021/12/01 13:52:48 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:52:53 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/01 13:52:53 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/01 13:52:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:52:58 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 13:53:03 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 13:53:08 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 13:53:13 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/01 13:53:13 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/01 13:53:13 No eventing-con process running
2021/12/01 13:53:13 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/01 13:53:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:53:25 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 13:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:53:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:53:36 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 13:53:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (308.82s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/01 13:53:46 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 13:53:46 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/01 13:53:46 No eventing-con process running
2021/12/01 13:53:46 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/01 13:53:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:53:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:54:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:54:00 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:54:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/01 13:54:06 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 13:54:11 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 13:54:16 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 13:54:21 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/01 13:54:21 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/01 13:54:21 Starting bucket ops to source bucket
2021/12/01 13:54:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:54:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:54:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:54:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:54:50 Waiting for dst bucket item count to get to: 5000 curr count: 4958
2021/12/01 13:54:55 Waiting for dst bucket item count to get to: 5000 curr count: 4958
2021/12/01 13:54:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:54:55 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:54:55 Pausing app: bucket_op_with_timer
2021/12/01 13:54:56 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:55:01 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/01 13:55:01 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/01 13:55:01 Resuming app: bucket_op_with_timer
2021/12/01 13:55:02 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:55:02 Starting bucket ops to source bucket
2021/12/01 13:55:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:55:19 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:55:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:55:29 Waiting for dst bucket item count to get to: 10000 curr count: 5334
2021/12/01 13:55:34 Waiting for dst bucket item count to get to: 10000 curr count: 5334
2021/12/01 13:55:34 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 13:55:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 13:55:39 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 13:55:44 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 13:55:49 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 13:55:54 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/01 13:55:54 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/01 13:55:54 No eventing-con process running
2021/12/01 13:55:54 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/01 13:55:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:56:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 13:56:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:56:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 13:56:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (160.99s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/01 13:56:27 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 13:56:27 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/01 13:56:27 No eventing-con process running
2021/12/01 13:56:27 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/01 13:56:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:56:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:56:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 13:56:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 13:56:41 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 13:56:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/01 13:56:47 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 13:56:52 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 13:56:57 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 13:57:02 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/01 13:57:02 App: TestPauseResumeWithEventingReb got deployed
2021/12/01 13:57:02 Starting bucket ops to source bucket
2021/12/01 13:57:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:57:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:57:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:57:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 13:57:31 Waiting for dst bucket item count to get to: 5000 curr count: 4071
2021/12/01 13:57:37 Waiting for dst bucket item count to get to: 5000 curr count: 4071
2021/12/01 13:57:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 13:57:37 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 13:57:37 Pausing app: bucket_op_with_timer
2021/12/01 13:57:37 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:57:42 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/01 13:57:42 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/01 13:57:42 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 13:57:42 Removing node(s): [] from the cluster
2021/12/01 13:57:47 Rebalance progress: 29.16666666666666
2021/12/01 13:57:52 Rebalance progress: 29.16666666666666
2021/12/01 13:57:57 Rebalance progress: 100
2021/12/01 13:57:57 Resuming app: bucket_op_with_timer
2021/12/01 13:57:58 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 13:57:58 Starting bucket ops to source bucket
2021/12/01 13:58:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:58:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:58:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:58:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/01 13:58:30 Waiting for dst bucket item count to get to: 10000 curr count: 5001
2021/12/01 13:58:35 Waiting for dst bucket item count to get to: 10000 curr count: 7501
2021/12/01 13:58:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 13:58:35 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 13:58:40 Rebalance progress: 29.16666666666666
2021/12/01 13:58:45 Rebalance progress: 29.16666666666666
2021/12/01 13:58:50 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/01 13:58:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 13:59:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 13:59:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 13:59:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 13:59:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:00:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:00:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:00:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:00:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:01:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:01:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:01:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 14:01:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cc05abbbdb2d5f85ad55fbeb0cedbabf
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 57250fff5d815f726026018074a4ac9b
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 959f64723635a4379387a1c9875d725a
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]


2021/12/01 14:02:10 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:02:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:02:16 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 14:02:21 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 14:02:26 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 14:02:31 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/01 14:02:31 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/01 14:02:31 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:02:31 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/01 14:02:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:02:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:02:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:02:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:02:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:02:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:03:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:03:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (406.98s)
=== RUN   TestBucketDeleteAfterPause
2021/12/01 14:03:14 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/01 14:03:14 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/01 14:03:14 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:03:14 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/01 14:03:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:03:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:03:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:03:28 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:03:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/01 14:03:34 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 14:03:39 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 14:03:44 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 14:03:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/01 14:03:49 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/01 14:03:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:03:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:04:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:04:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/01 14:04:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:04:09 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:04:09 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/01 14:04:09 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:04:14 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/01 14:04:14 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/01 14:04:14 handler_test.go:716: Deleting source bucket: default
2021/12/01 14:04:20 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/01 14:04:20 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/01 14:04:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/01 14:04:26 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/01 14:04:26 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:04:26 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/01 14:04:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:04:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:04:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:04:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:04:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:04:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:04:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (110.01s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/01 14:05:04 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 14:05:04 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/01 14:05:04 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:05:04 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/01 14:05:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:05:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:05:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:05:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:05:18 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:05:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/01 14:05:24 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 14:05:29 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 14:05:34 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 14:05:39 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/01 14:05:39 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/01 14:05:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:06:14 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/01 14:06:14 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:06:19 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/01 14:06:19 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/01 14:09:19 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/01 14:09:19 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/01 14:09:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/01 14:09:25 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 14:09:25 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:09:25 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/01 14:09:30 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 14:09:30 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:09:30 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2675 expected 5000
2021/12/01 14:09:35 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 14:09:35 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:09:35 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3540 expected 5000
2021/12/01 14:09:40 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/01 14:09:40 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:09:40 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/01 14:09:40 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/01 14:09:40 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:09:50 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 14:09:55 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 14:10:00 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/01 14:10:00 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/01 14:10:00 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:10:00 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/01 14:10:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:10:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:10:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:10:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:10:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:10:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.83s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/01 14:10:33 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 14:10:33 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/01 14:10:33 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:10:33 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/01 14:10:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:10:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:10:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:10:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:10:46 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:10:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 14:10:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:10:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:11:03 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:11:08 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:11:08 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/01 14:11:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:11:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:11:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1121
2021/12/01 14:11:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1121
2021/12/01 14:11:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:11:26 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:11:26 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/01 14:11:27 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:11:57 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/01 14:11:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 14:12:03 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:12:08 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:12:13 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:12:18 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/01 14:12:18 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/01 14:12:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:12:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:12:35 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:12:35 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/01 14:12:36 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:12:46 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 14:12:51 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 14:12:56 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/01 14:12:56 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/01 14:12:56 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:12:56 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/01 14:12:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:13:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:13:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:13:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.53s)
=== RUN   TestCPPWorkerCleanup
2021/12/01 14:13:19 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 14:13:19 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/01 14:13:19 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:13:19 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/01 14:13:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:13:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:13:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:13:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:13:32 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:13:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/01 14:13:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:13:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:13:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:13:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:13:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:14:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2681
2021/12/01 14:14:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4634
2021/12/01 14:14:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:14:05 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:14:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:14:10 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 14:14:15 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 14:14:20 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 14:14:25 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/01 14:14:25 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/01 14:14:25 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:14:25 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/01 14:14:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:14:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:14:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:14:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:14:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4186
2021/12/01 14:14:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1686
2021/12/01 14:14:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.69s)
=== RUN   TestWithUserXattrs
2021/12/01 14:15:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 14:15:33 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/01 14:15:33 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:15:33 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/01 14:15:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:15:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:15:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:15:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:15:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:15:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:15:51 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:15:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/01 14:15:57 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 14:16:02 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 14:16:07 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 14:16:12 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/01 14:16:12 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/01 14:16:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:16:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:16:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:16:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:16:26 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:16:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:16:31 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 14:16:36 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 14:16:41 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 14:16:46 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/01 14:16:46 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/01 14:16:46 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:16:46 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/01 14:16:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:16:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:16:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:16:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:17:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:17:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:17:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.11s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/01 14:17:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:17:25 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 14:17:25 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:17:25 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/01 14:17:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:17:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:17:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:17:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:17:38 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:17:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/01 14:17:44 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 14:17:49 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 14:17:54 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 14:17:59 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/01 14:17:59 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/01 14:17:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:18:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:18:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 228
2021/12/01 14:18:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2728
2021/12/01 14:18:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:18:18 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:18:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:18:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3950
2021/12/01 14:18:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3950
2021/12/01 14:18:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:18:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:18:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3984
2021/12/01 14:18:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1484
2021/12/01 14:18:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:18:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:19:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 12443
2021/12/01 14:19:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 18908
2021/12/01 14:19:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 22645
2021/12/01 14:19:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/01 14:19:28 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:19:28 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:19:33 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:19:38 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:19:43 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:19:48 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:19:48 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 14:19:48 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:19:53 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/01 14:19:53 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/01 14:19:53 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:19:53 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/01 14:19:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:20:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 14:20:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:20:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:20:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/01 14:20:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (181.79s)
=== RUN   TestMetaBucketDelete
2021/12/01 14:20:26 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 14:20:26 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 14:20:26 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:20:26 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/01 14:20:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:20:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:20:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:20:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:20:40 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:20:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 14:20:46 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 14:20:51 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 14:20:56 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 14:21:01 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/01 14:21:01 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/01 14:21:01 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/01 14:21:02 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/01 14:21:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 14:21:07 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 14:21:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/01 14:21:22 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/01 14:21:22 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:21:22 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/01 14:21:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:21:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:21:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:21:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (79.22s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/01 14:21:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:21:46 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:21:46 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:21:46 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 14:21:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:21:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:21:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:21:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:21:59 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:22:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 14:22:19 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/01 14:22:19 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/01 14:22:34 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:22:34 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:22:49 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:22:49 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:22:49 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:22:49 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 14:22:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:22:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:22:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:23:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.97s)
=== RUN   TestSourceBucketDelete
2021/12/01 14:23:12 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 14:23:12 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 14:23:13 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:23:13 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/01 14:23:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:23:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:23:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:23:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:23:26 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:23:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/01 14:23:32 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 14:23:37 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 14:23:42 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 14:23:47 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/01 14:23:47 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/01 14:23:47 handler_test.go:945: Deleting source bucket: default
2021/12/01 14:23:48 handler_test.go:947: Deleted source bucket: default
2021/12/01 14:23:53 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 14:23:53 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 14:24:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/01 14:24:08 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/01 14:24:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:24:08 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/01 14:24:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:24:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:24:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:24:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.06s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/01 14:24:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:24:32 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:24:32 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:24:32 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 14:24:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:24:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:24:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:24:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:24:45 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:24:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 14:25:05 handler_test.go:966: Deleting source bucket: default
2021/12/01 14:25:06 handler_test.go:968: Deleted source bucket: default
2021/12/01 14:25:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:25:21 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:25:36 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:25:36 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:25:36 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:25:36 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 14:25:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:25:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:25:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:25:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (88.11s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/01 14:26:00 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:26:00 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:26:00 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:26:00 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/01 14:26:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:26:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:26:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:26:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:26:13 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:26:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 14:26:33 handler_test.go:988: Deleting source bucket: default
2021/12/01 14:26:35 handler_test.go:990: Deleted source bucket: default
2021/12/01 14:26:35 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/01 14:26:35 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/01 14:26:50 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:26:50 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:27:05 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/01 14:27:05 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/01 14:27:05 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:27:05 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/01 14:27:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:27:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:27:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:27:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.49s)
=== RUN   TestUndeployDuringBootstrap
2021/12/01 14:27:28 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 14:27:28 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 14:27:28 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:27:28 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/01 14:27:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:27:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:27:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:27:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:27:41 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:27:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 14:27:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:27:54 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:27:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:27:55 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 14:27:55 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/01 14:27:55 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 14:28:00 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/01 14:28:05 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/01 14:28:10 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 14:28:15 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 14:28:20 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 14:28:20 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 14:28:20 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:28:45 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/01 14:28:45 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/01 14:28:45 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:28:45 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/01 14:28:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:28:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:28:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:28:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:29:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3076
2021/12/01 14:29:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 576
2021/12/01 14:29:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (114.68s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/01 14:29:18 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:29:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/01 14:29:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 14:29:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 14:29:34 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 14:29:39 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/01 14:29:39 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/01 14:29:39 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/01 14:29:39 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:29:39 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/01 14:29:44 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/01 14:29:44 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/01 14:29:44 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:29:44 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/01 14:29:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:29:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:29:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:29:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.74s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/01 14:30:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:30:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 14:30:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:30:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/01 14:30:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:30:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:30:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:30:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:30:21 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:30:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/01 14:30:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 14:30:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 14:30:37 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 14:30:42 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/01 14:30:42 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/01 14:30:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:31:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:31:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:31:23 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:31:28 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:31:33 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:31:33 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 14:31:33 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:33:18 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/01 14:33:18 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/01 14:33:18 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:33:18 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/01 14:33:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:33:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:33:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 14:33:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:33:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:33:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:33:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.39s)
=== RUN   TestUndeployWithKVFailover
2021/12/01 14:33:56 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 14:33:56 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/01 14:33:56 eventingSetup.go:46: No eventing-con process running
2021/12/01 14:33:56 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/01 14:33:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:34:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:34:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:34:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:34:09 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:34:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/01 14:34:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 14:34:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:34:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2950
2021/12/01 14:34:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2950
2021/12/01 14:34:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:34:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 14:34:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 14:34:58 Rebalance progress: 29.16666666666666
2021/12/01 14:35:03 Rebalance progress: 29.16666666666666
2021/12/01 14:35:08 Rebalance progress: 100
2021/12/01 14:35:08 Failing over: [127.0.0.1:19003]
2021/12/01 14:35:18 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/01 14:36:18 Removing node(s): [] from the cluster
2021/12/01 14:36:23 Rebalance progress: 29.16666666666666
2021/12/01 14:36:28 Rebalance progress: 29.16666666666666
2021/12/01 14:36:33 Rebalance progress: 100
2021/12/01 14:36:38 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 14:36:38 App: TestUndeployWithKVFailover got un-deployed
2021/12/01 14:36:38 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:36:43 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/01 14:36:43 App: TestUndeployWithKVFailover got un-deployed
2021/12/01 14:36:43 No eventing-con process running
2021/12/01 14:36:43 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/01 14:36:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:36:54 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:36:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:37:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.27s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/01 14:37:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 14:37:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/01 14:37:16 No eventing-con process running
2021/12/01 14:37:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/01 14:37:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:37:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:37:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:37:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:37:30 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:37:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/01 14:37:36 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 14:37:41 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 14:37:46 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 14:37:51 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/01 14:37:51 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/01 14:37:51 Starting bucket ops to source bucket
2021/12/01 14:38:11 Waiting for dst bucket item count to get to: 20000 curr count: 9805
2021/12/01 14:38:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/01 14:38:11 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:38:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:38:23 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/01 14:38:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:38:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:38:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:38:40 Waiting for dst bucket item count to get to: 0 curr count: 15057
2021/12/01 14:38:45 Waiting for dst bucket item count to get to: 0 curr count: 15057
2021/12/01 14:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:38:45 Starting bucket ops to source bucket
2021/12/01 14:38:58 Waiting for dst bucket item count to get to: 10000 curr count: 3566
2021/12/01 14:39:03 Waiting for dst bucket item count to get to: 10000 curr count: 3566
2021/12/01 14:39:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 14:39:03 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:39:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:39:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 14:39:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 14:39:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 14:39:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/01 14:39:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/01 14:39:23 No eventing-con process running
2021/12/01 14:39:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/01 14:39:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:39:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:39:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:39:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:39:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:39:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.34s)
=== RUN   TestTimerOverwrite
2021/12/01 14:39:57 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 14:39:57 App: TestTimerOverwrite got un-deployed
2021/12/01 14:39:57 No eventing-con process running
2021/12/01 14:39:57 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/01 14:39:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:40:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:40:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:40:10 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:40:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/01 14:40:16 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 14:40:21 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 14:40:26 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 14:40:31 Waiting for app: TestTimerOverwrite to get deployed
2021/12/01 14:40:31 App: TestTimerOverwrite got deployed
2021/12/01 14:40:41 Starting bucket ops to source bucket
2021/12/01 14:40:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:40:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 14:41:50 Waiting for dst bucket item count to get to: 5000 curr count: 2864
2021/12/01 14:41:55 Waiting for dst bucket item count to get to: 5000 curr count: 2864
2021/12/01 14:41:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:42:05 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:42:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:42:10 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 14:42:15 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 14:42:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 14:42:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/01 14:42:25 App: TestTimerOverwrite got un-deployed
2021/12/01 14:42:25 No eventing-con process running
2021/12/01 14:42:25 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/01 14:42:26 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:42:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:42:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:42:48 Waiting for dst bucket item count to get to: 0 curr count: 3997
2021/12/01 14:42:53 Waiting for dst bucket item count to get to: 0 curr count: 1497
2021/12/01 14:42:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.75s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/01 14:42:58 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 14:42:59 Removing node(s): [] from the cluster
2021/12/01 14:43:04 Rebalance progress: 100
2021/12/01 14:43:04 Failing over: [127.0.0.1:19003]
2021/12/01 14:43:09 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 14:43:09 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 14:43:09 No eventing-con process running
2021/12/01 14:43:09 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/01 14:43:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:43:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:43:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:43:22 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:43:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/01 14:43:23 Starting bucket ops to source bucket
2021/12/01 14:43:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:43:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:43:33 Context: Post to set retry counter, response code: 200 dump: 
2021/12/01 14:43:33 set retry counter, response &{[] }
2021/12/01 14:44:38 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 14:44:38 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 14:44:38 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:44:38 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/01 14:44:43 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/01 14:44:43 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/01 14:44:43 No eventing-con process running
2021/12/01 14:44:43 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/01 14:44:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:44:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:44:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:45:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:45:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:45:07 Removing node(s): [] from the cluster
2021/12/01 14:45:12 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.35s)
=== RUN   TestOnUpdateSrcMutation
2021/12/01 14:45:22 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 14:45:22 App: TestOnUpdateSrcMutation got un-deployed
2021/12/01 14:45:22 No eventing-con process running
2021/12/01 14:45:22 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/01 14:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:45:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:45:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:45:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:45:35 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:45:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/01 14:45:36 Starting bucket ops to source bucket
2021/12/01 14:46:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 14:46:03 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:46:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:46:08 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 14:46:13 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 14:46:18 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 14:46:23 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/01 14:46:23 App: TestOnUpdateSrcMutation got un-deployed
2021/12/01 14:46:23 No eventing-con process running
2021/12/01 14:46:23 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/01 14:46:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:46:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:46:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:46:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:46:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.60s)
=== RUN   TestOnDeleteSrcMutation
2021/12/01 14:46:51 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 14:46:51 App: TestOnDeleteSrcMutation got un-deployed
2021/12/01 14:46:51 No eventing-con process running
2021/12/01 14:46:51 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/01 14:46:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:46:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:46:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:47:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:47:04 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:47:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/01 14:47:06 Starting bucket ops to source bucket
2021/12/01 14:47:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:47:34 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:47:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:47:39 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 14:47:44 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 14:47:49 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 14:47:54 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/01 14:47:54 App: TestOnDeleteSrcMutation got un-deployed
2021/12/01 14:47:54 No eventing-con process running
2021/12/01 14:47:55 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/01 14:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:48:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:48:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:48:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:48:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.34s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/01 14:48:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 14:48:23 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/01 14:48:23 No eventing-con process running
2021/12/01 14:48:23 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/01 14:48:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:48:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:48:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:48:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:48:36 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:48:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 14:48:37 Starting bucket ops to source bucket
2021/12/01 14:49:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 14:49:14 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:49:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:49:19 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 14:49:24 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 14:49:29 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 14:49:34 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/01 14:49:34 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/01 14:49:34 No eventing-con process running
2021/12/01 14:49:34 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/01 14:49:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:49:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 14:49:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:49:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:49:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.91s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/01 14:50:03 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 14:50:03 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/01 14:50:03 No eventing-con process running
2021/12/01 14:50:03 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/01 14:50:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:50:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:50:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:50:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:50:16 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:50:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/01 14:50:17 Starting bucket ops to source bucket
2021/12/01 14:50:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 14:50:56 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 14:50:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:51:01 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 14:51:06 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 14:51:11 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 14:51:16 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/01 14:51:16 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/01 14:51:16 No eventing-con process running
2021/12/01 14:51:16 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/01 14:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:51:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 14:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:51:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:51:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.45s)
=== RUN   TestInterHandlerRecursion
2021/12/01 14:51:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:51:44 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 14:51:44 No eventing-con process running
2021/12/01 14:51:44 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/01 14:51:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:51:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:51:52 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:51:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:52:02 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:52:02 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 14:52:02 No eventing-con process running
2021/12/01 14:52:02 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 14:52:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:52:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:52:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:52:15 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:52:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/01 14:52:17 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:52:22 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:52:27 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:52:32 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:52:37 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:52:37 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/01 14:52:37 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:52:37 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/01 14:52:37 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/01 14:52:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:52:42 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:52:47 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:52:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:52:57 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:52:57 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 14:52:57 No eventing-con process running
2021/12/01 14:52:57 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/01 14:52:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:53:05 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:53:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:53:15 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:53:15 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 14:53:15 No eventing-con process running
2021/12/01 14:53:15 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 14:53:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:53:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:53:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.15s)
=== RUN   TestInterBucketRecursion
2021/12/01 14:53:33 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:53:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 14:53:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:53:34 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:53:34 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/01 14:53:39 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 14:53:44 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 14:53:49 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 14:53:54 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/01 14:53:54 App: TestInterBucketRecursion_function1 got deployed
2021/12/01 14:53:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:53:59 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:54:04 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:54:09 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:54:14 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:54:14 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/01 14:54:14 No eventing-con process running
2021/12/01 14:54:15 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/01 14:54:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:54:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:54:33 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/01 14:54:33 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/01 14:54:33 No eventing-con process running
2021/12/01 14:54:33 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/01 14:54:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:54:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:54:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.58s)
=== RUN   TestLargeHandler
2021/12/01 14:54:46 Changed force_compress value to false
2021/12/01 14:54:46 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:54:46 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/01 14:54:46 Changed force_compress value to true
2021/12/01 14:54:46 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:54:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/01 14:54:53 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 14:54:58 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 14:55:03 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 14:55:08 Waiting for app: TestLargeHandler_function to get deployed
2021/12/01 14:55:08 App: TestLargeHandler_function got deployed
2021/12/01 14:55:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:55:13 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 14:55:18 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 14:55:23 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/01 14:55:23 App: TestLargeHandler_function got un-deployed
2021/12/01 14:55:23 No eventing-con process running
2021/12/01 14:55:23 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/01 14:55:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:55:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:55:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (50.52s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/01 14:55:46 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:55:46 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 14:55:46 No eventing-con process running
2021/12/01 14:55:46 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/01 14:55:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:55:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:55:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:55:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:56:04 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:56:04 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 14:56:04 No eventing-con process running
2021/12/01 14:56:04 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/01 14:56:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:56:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:56:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:56:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:56:18 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:56:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/01 14:56:19 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:56:24 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:56:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:56:34 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:56:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/01 14:56:39 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/01 14:56:39 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:56:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/01 14:56:40 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:56:45 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 14:56:50 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 14:56:55 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 14:57:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/01 14:57:00 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/01 14:57:00 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/01 14:57:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:57:05 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:57:10 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:57:15 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:57:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/01 14:57:20 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/01 14:57:20 No eventing-con process running
2021/12/01 14:57:20 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/01 14:57:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:57:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:57:39 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:57:44 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:57:49 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:57:54 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/01 14:57:54 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/01 14:57:55 No eventing-con process running
2021/12/01 14:57:55 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/01 14:57:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:58:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:58:03 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:58:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.54s)
=== RUN   TestAllowInterBucketRecursion
2021/12/01 14:58:13 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:58:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 14:58:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:58:19 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 14:58:24 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 14:58:29 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 14:58:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/01 14:58:34 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/01 14:58:34 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 14:58:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 14:58:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/01 14:58:40 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 14:58:45 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 14:58:50 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 14:58:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/01 14:58:55 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/01 14:58:55 TestAllowInterBucketRecursion_function2 is deployed
2021/12/01 14:58:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:59:00 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:59:05 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:59:10 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:59:15 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/01 14:59:15 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/01 14:59:15 No eventing-con process running
2021/12/01 14:59:15 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/01 14:59:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:59:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:59:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:59:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:59:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 14:59:34 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 14:59:39 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 14:59:44 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 14:59:49 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/01 14:59:49 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/01 14:59:49 No eventing-con process running
2021/12/01 14:59:49 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/01 14:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 14:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 14:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:00:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.06s)
=== RUN   TestInterBucketRecursion2
2021/12/01 15:00:09 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:00:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:00:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:00:32 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:00:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 15:00:38 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 15:00:43 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 15:00:48 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 15:00:53 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/01 15:00:53 App: TestInterBucketRecursion2_function1 got deployed
2021/12/01 15:00:53 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:00:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 15:01:00 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 15:01:05 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 15:01:10 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 15:01:15 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/01 15:01:15 App: TestInterBucketRecursion2_function2 got deployed
2021/12/01 15:01:15 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:01:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 15:01:21 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 15:01:26 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 15:01:31 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 15:01:36 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/01 15:01:36 App: TestInterBucketRecursion2_function3 got deployed
2021/12/01 15:01:36 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:01:36 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/01 15:01:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:01:41 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 15:01:46 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 15:01:51 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 15:01:56 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/01 15:01:56 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/01 15:01:56 No eventing-con process running
2021/12/01 15:01:56 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/01 15:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:02:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:02:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:02:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:02:14 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 15:02:19 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 15:02:24 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 15:02:29 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/01 15:02:29 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/01 15:02:29 No eventing-con process running
2021/12/01 15:02:29 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/01 15:02:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:02:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:02:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:02:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:02:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:02:48 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 15:02:53 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 15:02:58 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 15:03:03 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/01 15:03:03 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/01 15:03:03 No eventing-con process running
2021/12/01 15:03:03 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/01 15:03:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:03:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:03:21 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/01 15:03:21 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/01 15:03:21 No eventing-con process running
2021/12/01 15:03:21 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/01 15:03:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:03:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:03:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:03:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.58s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/01 15:03:41 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/01 15:03:46 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/01 15:03:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:04:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:04:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:04:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:04:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:05:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:05:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:05:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/01 15:05:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/01 15:06:05 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:06:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 15:06:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 15:06:10 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/01 15:06:10 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:06:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 15:06:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 15:06:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/01 15:06:15 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:06:20 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/01 15:06:20 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/01 15:06:20 eventingSetup.go:46: No eventing-con process running
2021/12/01 15:06:20 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/01 15:06:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:06:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:06:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:06:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:06:38 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:06:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 15:06:44 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 15:06:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 15:06:54 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 15:06:59 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/01 15:06:59 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cc05abbbdb2d5f85ad55fbeb0cedbabf
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: 57250fff5d815f726026018074a4ac9b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 959f64723635a4379387a1c9875d725a
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]


2021/12/01 15:07:00 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:07:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 15:07:05 Rebalance progress: 29.16666666666666
2021/12/01 15:07:10 Rebalance progress: 29.16666666666666
2021/12/01 15:07:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: 57250fff5d815f726026018074a4ac9b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 959f64723635a4379387a1c9875d725a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: cc05abbbdb2d5f85ad55fbeb0cedbabf
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/01 15:07:15 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:07:15 handler_test.go:1674: Deleting source bucket: default
2021/12/01 15:07:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 15:07:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 15:07:23 Rebalance progress: 29.16666666666666
2021/12/01 15:07:28 Rebalance progress: 29.16666666666666
2021/12/01 15:07:33 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 900
	vb list dump: [0-2, 4-9, 12, 14-16, 18, 20-29, 31-35, 37-44, 46-82, 84-97, 99, 101-106, 108-121, 123-132, 134-161, 163-164, 166-176, 178-180, 182-187, 189-192, 194-198, 201-208, 210-218, 220-226, 228-230, 232, 234-247, 249-254, 256-257, 259-264, 266-272, 274-280, 282-292, 294-295, 297-318, 320-331, 333-335, 337-345, 347-348, 350-357, 359, 361-368, 370-380, 382-385, 387-388, 391-396, 398-400, 402-434, 436-448, 450-454, 456-457, 459, 461-463, 465, 467, 469-477, 479-492, 494, 498-503, 505-508, 511-571, 573-591, 593-610, 612-615, 617-619, 621-626, 628-635, 637-640, 642-644, 646-654, 656-665, 667-681, 683, 685-701, 705-715, 717-721, 723-728, 730-746, 748-766, 768-769, 771, 773-776, 779-789, 792-798, 800, 802, 804-809, 812-815, 817-828, 830-838, 840-853, 855-872, 875-877, 879-886, 888-890, 892-895, 898-906, 908-920, 922-930, 932, 934-938, 940-947, 949-958, 960-962, 964-968, 970, 972-973, 975-995, 997-1001, 1003-1005, 1007-1019, 1021-1023]
	stream status: stopped
	len: 124
	vb list dump: [3, 10-11, 13, 17, 19, 30, 36, 45, 83, 98, 100, 107, 122, 133, 162, 165, 177, 181, 188, 193, 199-200, 209, 219, 227, 231, 233, 248, 255, 258, 265, 273, 281, 293, 296, 319, 332, 336, 346, 349, 358, 360, 369, 381, 386, 389-390, 397, 401, 435, 449, 455, 458, 460, 464, 466, 468, 478, 493, 495-497, 504, 509-510, 572, 592, 611, 616, 620, 627, 636, 641, 645, 655, 666, 682, 684, 702-704, 716, 722, 729, 747, 767, 770, 772, 777-778, 790-791, 799, 801, 803, 810-811, 816, 829, 839, 854, 873-874, 878, 887, 891, 896-897, 907, 921, 931, 933, 939, 948, 959, 963, 969, 971, 974, 996, 1002, 1006, 1020]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: cc05abbbdb2d5f85ad55fbeb0cedbabf
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-2, 4-9, 12, 14-16, 18, 20-29, 31-35, 37-44, 46-82, 84-97, 99, 101-106, 108-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-121, 123-132, 134-161, 163-164, 166-176, 178-180, 182-187, 189-192, 194-198, 201-208, 210-218, 220-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-230, 232, 234-247, 249-254, 256-257, 259-264, 266-272, 274-280, 282-292, 294-295, 297-318, 320-331, 333-335, 337-341]
Producer node: 172.31.2.134:9300	Node UUID: 57250fff5d815f726026018074a4ac9b
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [911-920, 922-930, 932, 934-938, 940-947, 949-958, 960-962, 964-968, 970, 972-973, 975-995, 997-1001, 1003-1005, 1007-1019, 1021-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [683, 685-701, 705-715, 717-721, 723-728, 730-746, 748-766, 768-769, 771, 773-776, 779-789, 792-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 97
	v: [797-798, 800, 802, 804-809, 812-815, 817-828, 830-838, 840-853, 855-872, 875-877, 879-886, 888-890, 892-895, 898-906, 908-910]
Producer node: 127.0.0.1:9303	Node UUID: 959f64723635a4379387a1c9875d725a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-345, 347-348, 350-357, 359, 361-368, 370-380, 382-385, 387-388, 391-396, 398-400, 402-434, 436-448, 450-454]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456-457, 459, 461-463, 465, 467, 469-477, 479-492, 494, 498-503, 505-508, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-571, 573-591, 593-610, 612-615, 617-619, 621-626, 628-635, 637-640, 642-644, 646-654, 656-665, 667-681]


2021/12/01 15:07:33 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:07:38 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/01 15:07:38 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/01 15:07:43 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/01 15:07:43 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/01 15:07:43 eventingSetup.go:46: No eventing-con process running
2021/12/01 15:07:43 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/01 15:07:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:07:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:07:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:07:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:07:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 15:07:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 15:08:02 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (268.12s)
=== RUN   TestN1QLRecursion
2021/12/01 15:08:05 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 15:08:05 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 15:08:05 Removing node(s): [] from the cluster
2021/12/01 15:08:10 Rebalance progress: 100
2021/12/01 15:08:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:08:22 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:08:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:08:33 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:08:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 15:08:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 15:08:39 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:08:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:08:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:08:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:08:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:09:04 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:09:09 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/01 15:09:09 App: TestN1QLRecursion_function1 got deployed
2021/12/01 15:09:09 TestN1QLRecursion_function1 is deployed
2021/12/01 15:09:09 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:09:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 15:09:10 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/01 15:09:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 15:09:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 15:09:25 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 15:09:30 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/01 15:09:30 App: TestN1QLRecursion_function3 got deployed
2021/12/01 15:09:30 TestN1QLRecursion_function3 is deployed
2021/12/01 15:09:30 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:09:30 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/01 15:09:30 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/01 15:09:30 Success: TestN1QLRecursion_function2 is not deployed
2021/12/01 15:09:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:09:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 15:09:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 15:09:46 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 15:09:51 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/01 15:09:51 App: TestN1QLRecursion_function1 got un-deployed
2021/12/01 15:09:51 No eventing-con process running
2021/12/01 15:09:51 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/01 15:09:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:09:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:10:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:10:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:10:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:10:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 15:10:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 15:10:20 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 15:10:25 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/01 15:10:25 App: TestN1QLRecursion_function3 got un-deployed
2021/12/01 15:10:25 No eventing-con process running
2021/12/01 15:10:25 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/01 15:10:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:10:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:10:40 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 15:10:45 Rebalance progress: 100
--- PASS: TestN1QLRecursion (160.53s)
=== RUN   TestN1QLAllowRecursion
2021/12/01 15:10:45 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 15:10:45 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 15:10:45 Removing node(s): [] from the cluster
2021/12/01 15:10:50 Rebalance progress: 100
2021/12/01 15:10:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:11:02 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:11:13 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/01 15:11:14 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/01 15:11:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 15:11:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 15:11:29 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 15:11:34 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/01 15:11:34 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/01 15:11:34 TestN1QLAllowRecursion_function1 is deployed
2021/12/01 15:11:34 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:11:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/01 15:11:36 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/01 15:11:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 15:11:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 15:11:51 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 15:11:56 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/01 15:11:56 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/01 15:11:56 TestN1QLAllowRecursion_function3 is deployed
2021/12/01 15:11:56 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:11:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/01 15:11:57 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/01 15:12:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 15:12:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 15:12:12 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 15:12:17 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/01 15:12:17 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/01 15:12:17 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/01 15:12:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:12:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 15:12:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 15:12:32 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 15:12:37 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/01 15:12:37 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/01 15:12:37 No eventing-con process running
2021/12/01 15:12:37 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/01 15:12:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:12:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:12:46 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:12:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:12:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:12:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 15:13:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 15:13:06 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 15:13:11 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/01 15:13:11 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/01 15:13:11 No eventing-con process running
2021/12/01 15:13:11 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/01 15:13:14 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:13:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:13:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:13:31 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 15:13:36 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 15:13:41 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 15:13:46 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/01 15:13:46 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/01 15:13:46 No eventing-con process running
2021/12/01 15:13:46 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/01 15:13:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:14:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:14:01 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 15:14:06 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (201.41s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/01 15:14:06 Sleeping for some time
2021/12/01 15:14:16 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 15:14:16 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/01 15:14:16 No eventing-con process running
2021/12/01 15:14:16 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/01 15:14:18 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:14:25 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:14:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:14:30 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:14:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 15:14:31 Starting bucket ops to source bucket
2021/12/01 15:15:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 15:15:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 15:15:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 15:15:26 Starting bucket ops to source bucket
2021/12/01 15:15:27 Starting bucket ops to source bucket
2021/12/01 15:16:33 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/01 15:16:38 Waiting for dst bucket item count to get to: 7000 curr count: 6355
2021/12/01 15:16:43 Waiting for dst bucket item count to get to: 7000 curr count: 6355
2021/12/01 15:16:48 Waiting for dst bucket item count to get to: 7000 curr count: 6355
2021/12/01 15:16:48 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/01 15:16:48 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:16:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:16:53 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 15:16:58 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 15:17:03 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 15:17:08 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/01 15:17:08 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/01 15:17:08 No eventing-con process running
2021/12/01 15:17:08 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/01 15:17:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:17:20 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/01 15:17:25 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/01 15:17:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:17:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:17:37 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/01 15:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (210.18s)
=== RUN   TestTimerOverWriteSameReference
2021/12/01 15:17:42 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 15:17:42 App: TestTimerOverWriteSameReference got un-deployed
2021/12/01 15:17:42 No eventing-con process running
2021/12/01 15:17:42 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/01 15:17:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:17:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:17:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:17:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:17:55 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:17:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/01 15:18:02 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 15:18:07 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 15:18:12 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 15:18:17 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/01 15:18:17 App: TestTimerOverWriteSameReference got deployed
2021/12/01 15:18:27 Starting bucket ops to source bucket
2021/12/01 15:19:28 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/01 15:19:38 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:19:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:19:43 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 15:19:48 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 15:19:53 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 15:19:58 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/01 15:19:58 App: TestTimerOverWriteSameReference got un-deployed
2021/12/01 15:19:58 No eventing-con process running
2021/12/01 15:19:58 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/01 15:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:20:10 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 15:20:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 15:20:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:20:27 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/01 15:20:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.91s)
=== RUN   TestCancelTimerBucketop
2021/12/01 15:20:32 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 15:20:32 App: TestCancelTimerBucketop got un-deployed
2021/12/01 15:20:32 No eventing-con process running
2021/12/01 15:20:32 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/01 15:20:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:20:45 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:20:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/01 15:20:51 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 15:20:56 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 15:21:01 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 15:21:07 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/01 15:21:07 App: TestCancelTimerBucketop got deployed
2021/12/01 15:21:07 Starting bucket ops to source bucket
2021/12/01 15:21:14 Starting bucket ops to source bucket
2021/12/01 15:22:16 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 15:22:16 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:22:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:22:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 15:22:26 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 15:22:31 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 15:22:36 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/01 15:22:36 App: TestCancelTimerBucketop got un-deployed
2021/12/01 15:22:36 No eventing-con process running
2021/12/01 15:22:36 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/01 15:22:38 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:22:48 Waiting for dst bucket item count to get to: 0 curr count: 585
2021/12/01 15:22:53 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/01 15:22:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:22:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:23:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.23s)
=== RUN   TestJSExpiryDate
2021/12/01 15:23:10 Starting bucket ops to source bucket
2021/12/01 15:23:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 15:23:23 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/01 15:23:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/01 15:23:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:23:54 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:23:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:23:59 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 15:24:04 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 15:24:09 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 15:24:14 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/01 15:24:14 App: TestJSExpiryDate got un-deployed
2021/12/01 15:24:14 No eventing-con process running
2021/12/01 15:24:14 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/01 15:24:16 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:24:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.74s)
=== RUN   TestBinaryDoc
2021/12/01 15:24:34 Starting bucket ops to source bucket
2021/12/01 15:24:34 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:24:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/01 15:24:35 Starting bucket ops to source bucket
2021/12/01 15:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:24:35 Deleting source bucket docs now
2021/12/01 15:24:35 Starting bucket ops to source bucket
2021/12/01 15:25:12 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/01 15:25:17 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/01 15:25:17 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/01 15:25:17 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:25:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:25:22 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 15:25:27 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 15:25:32 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 15:25:37 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/01 15:25:37 App: TestBinaryDoc got un-deployed
2021/12/01 15:25:37 No eventing-con process running
2021/12/01 15:25:37 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/01 15:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:25:49 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 15:25:54 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/01 15:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:25:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:26:06 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/01 15:26:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.36s)
=== RUN   TestConstantBindings
2021/12/01 15:26:06 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:26:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/01 15:26:07 Starting bucket ops to source bucket
2021/12/01 15:26:13 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 15:26:18 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 15:26:23 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/01 15:26:28 Waiting for dst bucket item count to get to: 6000 curr count: 650
2021/12/01 15:26:33 Waiting for dst bucket item count to get to: 6000 curr count: 650
2021/12/01 15:26:33 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/01 15:26:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:26:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:26:38 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 15:26:43 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 15:26:48 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 15:26:53 Waiting for app: TestConstantBindings to get un-deployed
2021/12/01 15:26:53 App: TestConstantBindings got un-deployed
2021/12/01 15:26:53 No eventing-con process running
2021/12/01 15:26:53 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/01 15:26:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:27:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:27:06 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:27:17 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/01 15:27:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.63s)
=== RUN   TestCodeInsight
2021/12/01 15:27:22 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 15:27:22 App: TestCodeInsight got un-deployed
2021/12/01 15:27:22 No eventing-con process running
2021/12/01 15:27:22 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/01 15:27:23 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:27:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:27:30 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:27:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:27:35 Starting bucket ops to source bucket
2021/12/01 15:27:36 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:27:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/01 15:27:42 Waiting for app: TestCodeInsight to get deployed
2021/12/01 15:27:47 Waiting for app: TestCodeInsight to get deployed
2021/12/01 15:27:52 Waiting for app: TestCodeInsight to get deployed
2021/12/01 15:27:57 Waiting for app: TestCodeInsight to get deployed
2021/12/01 15:27:57 App: TestCodeInsight got deployed
2021/12/01 15:28:02 Waiting for dst bucket item count to get to: 1000 curr count: 180
2021/12/01 15:28:02 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 15:28:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:28:07 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 15:28:12 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 15:28:17 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 15:28:22 Waiting for app: TestCodeInsight to get un-deployed
2021/12/01 15:28:22 App: TestCodeInsight got un-deployed
2021/12/01 15:28:22 No eventing-con process running
2021/12/01 15:28:22 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/01 15:28:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:28:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:28:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 15:28:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:28:41 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:28:51 Waiting for dst bucket item count to get to: 0 curr count: 914
2021/12/01 15:28:56 Waiting for dst bucket item count to get to: 0 curr count: 914
2021/12/01 15:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.28s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12791.183s

Functional tests took 3 hours, 33 minutes and 15 seconds.

N1ql tests

2021/12/01 15:29:07 Started n1ql
2021/12/01 15:29:17 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/01 15:29:18 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/01 15:29:18 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 15:29:19 Removing node(s): [] from the cluster
2021/12/01 15:29:24 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/01 15:29:29 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/01 15:29:29 App: TestRecursiveMutationN1QL got un-deployed
2021/12/01 15:29:29 No eventing-con process running
2021/12/01 15:29:29 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/01 15:29:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:29:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:29:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:29:42 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:29:42 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.61s)
=== RUN   TestFlexReset1
2021/12/01 15:29:47 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 15:29:47 App: TestFlexReset1 got un-deployed
2021/12/01 15:29:47 No eventing-con process running
2021/12/01 15:29:47 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/01 15:29:49 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:29:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:29:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:30:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:30:01 Starting bucket ops to source bucket
2021/12/01 15:30:02 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:30:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/01 15:30:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 15:30:13 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 15:30:18 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 15:30:23 Waiting for app: TestFlexReset1 to get deployed
2021/12/01 15:30:23 App: TestFlexReset1 got deployed
2021/12/01 15:30:28 Waiting for dst bucket item count to get to: 1000 curr count: 769
2021/12/01 15:30:33 Waiting for dst bucket item count to get to: 1000 curr count: 769
2021/12/01 15:30:33 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 15:30:33 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:30:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:30:38 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 15:30:43 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 15:30:48 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 15:30:53 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/01 15:30:53 App: TestFlexReset1 got un-deployed
2021/12/01 15:30:53 No eventing-con process running
2021/12/01 15:30:53 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/01 15:30:55 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:31:05 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:31:07 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:31:17 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:31:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.65s)
=== RUN   TestFlexReset2
2021/12/01 15:31:22 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 15:31:22 App: TestFlexReset2 got un-deployed
2021/12/01 15:31:22 No eventing-con process running
2021/12/01 15:31:22 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/01 15:31:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:31:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:31:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:31:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:31:36 Starting bucket ops to source bucket
2021/12/01 15:31:36 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:31:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/01 15:31:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 15:31:48 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 15:31:53 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 15:31:58 Waiting for app: TestFlexReset2 to get deployed
2021/12/01 15:31:58 App: TestFlexReset2 got deployed
2021/12/01 15:32:03 Waiting for dst bucket item count to get to: 1000 curr count: 31
2021/12/01 15:32:03 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/01 15:32:03 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:32:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:32:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 15:32:13 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 15:32:18 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 15:32:23 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/01 15:32:23 App: TestFlexReset2 got un-deployed
2021/12/01 15:32:23 No eventing-con process running
2021/12/01 15:32:23 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/01 15:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:32:34 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:32:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:32:36 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:32:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/01 15:32:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.26s)
=== RUN   TestN1QLLabelledBreak
2021/12/01 15:32:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 15:32:51 App: TestN1QLLabelledBreak got un-deployed
2021/12/01 15:32:51 No eventing-con process running
2021/12/01 15:32:51 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/01 15:32:53 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:32:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:33:00 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:33:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:33:05 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:33:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/01 15:33:11 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 15:33:16 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 15:33:21 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 15:33:26 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/01 15:33:26 App: TestN1QLLabelledBreak got deployed
2021/12/01 15:33:26 Starting bucket ops to source bucket
2021/12/01 15:33:35 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/01 15:33:40 Waiting for dst bucket item count to get to: 10000 curr count: 278
2021/12/01 15:33:45 Waiting for dst bucket item count to get to: 10000 curr count: 278
2021/12/01 15:33:50 Waiting for dst bucket item count to get to: 10000 curr count: 3775
2021/12/01 15:33:55 Waiting for dst bucket item count to get to: 10000 curr count: 3775
2021/12/01 15:34:01 Waiting for dst bucket item count to get to: 10000 curr count: 7822
2021/12/01 15:34:06 Waiting for dst bucket item count to get to: 10000 curr count: 7822
2021/12/01 15:34:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 15:34:06 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:34:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:34:11 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 15:34:16 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 15:34:21 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 15:34:26 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/01 15:34:26 App: TestN1QLLabelledBreak got un-deployed
2021/12/01 15:34:26 No eventing-con process running
2021/12/01 15:34:26 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/01 15:34:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:34:38 Waiting for dst bucket item count to get to: 0 curr count: 2660
2021/12/01 15:34:43 Waiting for dst bucket item count to get to: 0 curr count: 160
2021/12/01 15:34:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:34:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:34:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 15:34:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.23s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/01 15:35:00 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 15:35:00 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/01 15:35:00 No eventing-con process running
2021/12/01 15:35:00 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/01 15:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:35:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:35:08 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:35:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:35:13 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:35:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/01 15:35:19 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 15:35:24 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 15:35:29 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 15:35:34 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/01 15:35:34 App: TestN1QLUnlabelledBreak got deployed
2021/12/01 15:35:34 Starting bucket ops to source bucket
2021/12/01 15:35:44 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/01 15:35:49 Waiting for dst bucket item count to get to: 10000 curr count: 316
2021/12/01 15:35:54 Waiting for dst bucket item count to get to: 10000 curr count: 963
2021/12/01 15:35:59 Waiting for dst bucket item count to get to: 10000 curr count: 3600
2021/12/01 15:36:04 Waiting for dst bucket item count to get to: 10000 curr count: 6507
2021/12/01 15:36:09 Waiting for dst bucket item count to get to: 10000 curr count: 8554
2021/12/01 15:36:09 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 15:36:09 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:36:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:36:14 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 15:36:19 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 15:36:24 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 15:36:29 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/01 15:36:29 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/01 15:36:29 No eventing-con process running
2021/12/01 15:36:29 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/01 15:36:31 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:36:41 Waiting for dst bucket item count to get to: 0 curr count: 4529
2021/12/01 15:36:46 Waiting for dst bucket item count to get to: 0 curr count: 4529
2021/12/01 15:36:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:36:48 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:36:58 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 15:36:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.08s)
=== RUN   TestN1QLThrowStatement
2021/12/01 15:37:03 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 15:37:03 App: TestN1QLThrowStatement got un-deployed
2021/12/01 15:37:03 No eventing-con process running
2021/12/01 15:37:03 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/01 15:37:04 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:37:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:37:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:37:16 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:37:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/01 15:37:22 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 15:37:27 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 15:37:32 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 15:37:37 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/01 15:37:37 App: TestN1QLThrowStatement got deployed
2021/12/01 15:37:37 Starting bucket ops to source bucket
2021/12/01 15:37:47 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/01 15:37:52 Waiting for dst bucket item count to get to: 10000 curr count: 123
2021/12/01 15:37:57 Waiting for dst bucket item count to get to: 10000 curr count: 123
2021/12/01 15:38:02 Waiting for dst bucket item count to get to: 10000 curr count: 3222
2021/12/01 15:38:07 Waiting for dst bucket item count to get to: 10000 curr count: 3222
2021/12/01 15:38:12 Waiting for dst bucket item count to get to: 10000 curr count: 7287
2021/12/01 15:38:17 Waiting for dst bucket item count to get to: 10000 curr count: 7287
2021/12/01 15:38:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/01 15:38:17 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:38:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:38:22 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 15:38:27 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 15:38:32 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 15:38:37 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/01 15:38:37 App: TestN1QLThrowStatement got un-deployed
2021/12/01 15:38:37 No eventing-con process running
2021/12/01 15:38:37 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/01 15:38:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:38:49 Waiting for dst bucket item count to get to: 0 curr count: 4839
2021/12/01 15:38:54 Waiting for dst bucket item count to get to: 0 curr count: 2339
2021/12/01 15:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:38:56 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:39:06 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/01 15:39:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.11s)
=== RUN   TestN1QLNestedForLoop
2021/12/01 15:39:11 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 15:39:11 App: TestN1QLNestedForLoop got un-deployed
2021/12/01 15:39:11 No eventing-con process running
2021/12/01 15:39:11 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/01 15:39:13 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:39:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:39:19 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:39:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:39:24 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:39:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/01 15:39:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 15:39:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 15:39:41 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 15:39:46 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/01 15:39:46 App: TestN1QLNestedForLoop got deployed
2021/12/01 15:39:46 Starting bucket ops to source bucket
2021/12/01 15:39:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 15:40:00 Waiting for dst bucket item count to get to: 5000 curr count: 187
2021/12/01 15:40:05 Waiting for dst bucket item count to get to: 5000 curr count: 187
2021/12/01 15:40:10 Waiting for dst bucket item count to get to: 5000 curr count: 2204
2021/12/01 15:40:15 Waiting for dst bucket item count to get to: 5000 curr count: 2204
2021/12/01 15:40:20 Waiting for dst bucket item count to get to: 5000 curr count: 4572
2021/12/01 15:40:25 Waiting for dst bucket item count to get to: 5000 curr count: 4572
2021/12/01 15:40:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 15:40:25 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:40:30 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 15:40:35 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 15:40:40 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 15:40:45 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/01 15:40:45 App: TestN1QLNestedForLoop got un-deployed
2021/12/01 15:40:45 No eventing-con process running
2021/12/01 15:40:45 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/01 15:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:40:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 15:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:41:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 15:41:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 15:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.29s)
=== RUN   TestN1QLAutoClose
2021/12/01 15:41:19 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 15:41:19 App: TestN1QLAutoClose got un-deployed
2021/12/01 15:41:19 No eventing-con process running
2021/12/01 15:41:19 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/01 15:41:21 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:41:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:41:28 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:41:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:41:33 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:41:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/01 15:41:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 15:41:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 15:41:49 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 15:41:54 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/01 15:41:54 App: TestN1QLAutoClose got deployed
2021/12/01 15:41:54 Starting bucket ops to source bucket
2021/12/01 15:41:59 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/01 15:42:04 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/01 15:42:09 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/01 15:42:14 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/01 15:42:20 Waiting for dst bucket item count to get to: 500 curr count: 30
2021/12/01 15:42:25 Waiting for dst bucket item count to get to: 500 curr count: 41
2021/12/01 15:42:30 Waiting for dst bucket item count to get to: 500 curr count: 52
2021/12/01 15:42:35 Waiting for dst bucket item count to get to: 500 curr count: 63
2021/12/01 15:42:40 Waiting for dst bucket item count to get to: 500 curr count: 75
2021/12/01 15:42:45 Waiting for dst bucket item count to get to: 500 curr count: 86
2021/12/01 15:42:50 Waiting for dst bucket item count to get to: 500 curr count: 97
2021/12/01 15:42:55 Waiting for dst bucket item count to get to: 500 curr count: 106
2021/12/01 15:43:00 Waiting for dst bucket item count to get to: 500 curr count: 119
2021/12/01 15:43:05 Waiting for dst bucket item count to get to: 500 curr count: 119
2021/12/01 15:43:10 Waiting for dst bucket item count to get to: 500 curr count: 127
2021/12/01 15:43:15 Waiting for dst bucket item count to get to: 500 curr count: 138
2021/12/01 15:43:20 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/01 15:43:25 Waiting for dst bucket item count to get to: 500 curr count: 161
2021/12/01 15:43:30 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/01 15:43:35 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/01 15:43:40 Waiting for dst bucket item count to get to: 500 curr count: 205
2021/12/01 15:43:45 Waiting for dst bucket item count to get to: 500 curr count: 205
2021/12/01 15:43:50 Waiting for dst bucket item count to get to: 500 curr count: 227
2021/12/01 15:43:55 Waiting for dst bucket item count to get to: 500 curr count: 227
2021/12/01 15:44:00 Waiting for dst bucket item count to get to: 500 curr count: 248
2021/12/01 15:44:05 Waiting for dst bucket item count to get to: 500 curr count: 248
2021/12/01 15:44:10 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/01 15:44:15 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/01 15:44:20 Waiting for dst bucket item count to get to: 500 curr count: 290
2021/12/01 15:44:25 Waiting for dst bucket item count to get to: 500 curr count: 290
2021/12/01 15:44:30 Waiting for dst bucket item count to get to: 500 curr count: 311
2021/12/01 15:44:35 Waiting for dst bucket item count to get to: 500 curr count: 311
2021/12/01 15:44:40 Waiting for dst bucket item count to get to: 500 curr count: 333
2021/12/01 15:44:45 Waiting for dst bucket item count to get to: 500 curr count: 333
2021/12/01 15:44:50 Waiting for dst bucket item count to get to: 500 curr count: 354
2021/12/01 15:44:55 Waiting for dst bucket item count to get to: 500 curr count: 354
2021/12/01 15:45:00 Waiting for dst bucket item count to get to: 500 curr count: 375
2021/12/01 15:45:05 Waiting for dst bucket item count to get to: 500 curr count: 375
2021/12/01 15:45:10 Waiting for dst bucket item count to get to: 500 curr count: 395
2021/12/01 15:45:15 Waiting for dst bucket item count to get to: 500 curr count: 395
2021/12/01 15:45:20 Waiting for dst bucket item count to get to: 500 curr count: 416
2021/12/01 15:45:25 Waiting for dst bucket item count to get to: 500 curr count: 429
2021/12/01 15:45:30 Waiting for dst bucket item count to get to: 500 curr count: 438
2021/12/01 15:45:35 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/01 15:45:40 Waiting for dst bucket item count to get to: 500 curr count: 460
2021/12/01 15:45:45 Waiting for dst bucket item count to get to: 500 curr count: 471
2021/12/01 15:45:50 Waiting for dst bucket item count to get to: 500 curr count: 483
2021/12/01 15:45:55 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/01 15:45:55 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/01 15:45:55 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:45:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:46:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 15:46:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 15:46:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 15:46:15 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/01 15:46:15 App: TestN1QLAutoClose got un-deployed
2021/12/01 15:46:15 No eventing-con process running
2021/12/01 15:46:15 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/01 15:46:17 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:46:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/01 15:46:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:46:29 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:46:39 Waiting for dst bucket item count to get to: 0 curr count: 456
2021/12/01 15:46:44 Waiting for dst bucket item count to get to: 0 curr count: 206
2021/12/01 15:46:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.71s)
=== RUN   TestN1QLPosParams
2021/12/01 15:46:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 15:46:49 App: TestN1QLPosParams got un-deployed
2021/12/01 15:46:49 No eventing-con process running
2021/12/01 15:46:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/01 15:46:51 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:46:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:47:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:47:02 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:47:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/01 15:47:09 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 15:47:14 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 15:47:19 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 15:47:24 Waiting for app: TestN1QLPosParams to get deployed
2021/12/01 15:47:24 App: TestN1QLPosParams got deployed
2021/12/01 15:47:24 Starting bucket ops to source bucket
2021/12/01 15:47:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/01 15:47:38 Waiting for dst bucket item count to get to: 5000 curr count: 582
2021/12/01 15:47:43 Waiting for dst bucket item count to get to: 5000 curr count: 1773
2021/12/01 15:47:48 Waiting for dst bucket item count to get to: 5000 curr count: 3691
2021/12/01 15:47:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 15:47:48 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:47:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:47:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 15:47:58 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 15:48:03 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 15:48:08 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/01 15:48:08 App: TestN1QLPosParams got un-deployed
2021/12/01 15:48:08 No eventing-con process running
2021/12/01 15:48:08 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/01 15:48:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:48:20 Waiting for dst bucket item count to get to: 0 curr count: 3311
2021/12/01 15:48:25 Waiting for dst bucket item count to get to: 0 curr count: 3311
2021/12/01 15:48:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:48:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 15:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.16s)
=== RUN   TestN1QLExhaustConnPool
2021/12/01 15:48:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 15:48:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/01 15:48:42 No eventing-con process running
2021/12/01 15:48:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/01 15:48:44 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:48:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:48:50 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:48:56 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:48:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/01 15:49:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 15:49:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 15:49:12 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 15:49:17 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/01 15:49:17 App: TestN1QLExhaustConnPool got deployed
2021/12/01 15:49:17 Starting bucket ops to source bucket
2021/12/01 15:49:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/01 15:49:27 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/01 15:49:32 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/01 15:49:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/01 15:49:32 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:49:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:49:37 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 15:49:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 15:49:47 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 15:49:52 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/01 15:49:52 App: TestN1QLExhaustConnPool got un-deployed
2021/12/01 15:49:52 No eventing-con process running
2021/12/01 15:49:52 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/01 15:49:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:50:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/01 15:50:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/01 15:50:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:50:11 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:50:21 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/01 15:50:26 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/01 15:50:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (109.00s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/01 15:50:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 15:50:31 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/01 15:50:31 No eventing-con process running
2021/12/01 15:50:31 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/01 15:50:33 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:50:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:50:45 Starting bucket ops to source bucket
2021/12/01 15:51:02 Starting bucket ops to source bucket
2021/12/01 15:51:20 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 10576
2021/12/01 15:51:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/01 15:51:26 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 15:51:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 15:51:36 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 15:51:41 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/01 15:51:41 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/01 15:51:41 Starting bucket ops to source bucket
2021/12/01 15:51:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:51:51 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:51:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:32 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/01 15:52:32 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/01 15:52:32 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:52:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:52:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 15:52:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 15:52:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 15:52:52 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/01 15:52:52 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/01 15:52:52 No eventing-con process running
2021/12/01 15:52:52 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/01 15:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:53:04 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/01 15:53:09 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/01 15:53:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:53:10 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:53:20 Waiting for dst bucket item count to get to: 0 curr count: 24233
2021/12/01 15:53:25 Waiting for dst bucket item count to get to: 0 curr count: 24233
2021/12/01 15:53:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.57s)
=== RUN   TestN1QLQueryConsistency
2021/12/01 15:53:30 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 15:53:30 App: n1qlquery_consistency got un-deployed
2021/12/01 15:53:30 No eventing-con process running
2021/12/01 15:53:30 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/01 15:53:32 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:53:39 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:53:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:53:44 Starting bucket ops to source bucket
2021/12/01 15:53:47 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:53:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/01 15:53:53 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 15:53:58 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 15:54:03 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 15:54:08 Waiting for app: n1qlquery_consistency to get deployed
2021/12/01 15:54:08 App: n1qlquery_consistency got deployed
2021/12/01 15:54:14 Waiting for dst bucket item count to get to: 5000 curr count: 1
2021/12/01 15:54:19 Waiting for dst bucket item count to get to: 5000 curr count: 4384
2021/12/01 15:54:24 Waiting for dst bucket item count to get to: 5000 curr count: 4384
2021/12/01 15:54:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/01 15:54:24 http call resp: Get http://127.0.0.1:9301/api/v1/stats: dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/01 15:54:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 15:54:29 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 15:54:34 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 15:54:39 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 15:54:44 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/01 15:54:44 App: n1qlquery_consistency got un-deployed
2021/12/01 15:54:44 No eventing-con process running
2021/12/01 15:54:44 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/01 15:54:45 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:54:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/01 15:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/01 15:55:07 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/01 15:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.96s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1555.360s

N1ql tests took 0 hours, 26 minutes and 0 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/01 15:56:43 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/01 15:57:02 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 15:57:02 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/01 15:57:02 No eventing-con process running
2021/12/01 15:57:02 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/01 15:57:03 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:57:08 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 15:57:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 15:57:10 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:57:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 15:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:57:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 15:57:20 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 15:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 15:57:26 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 15:57:31 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 15:57:36 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 15:57:41 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/01 15:57:41 App: TestDuplicateWithEventingRebIn got deployed
2021/12/01 15:57:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 15:57:41 Starting bucket ops to source bucket
2021/12/01 15:57:56 Removing node(s): [] from the cluster
2021/12/01 15:58:01 Rebalance progress: 25
2021/12/01 15:58:06 Rebalance progress: 25
2021/12/01 15:58:11 Rebalance progress: 25
2021/12/01 15:58:16 Rebalance progress: 25
2021/12/01 15:58:21 Rebalance progress: 75
2021/12/01 15:58:26 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:58:26 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:58:26 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 15:58:26 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9302/api/v1/stats: dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 15:58:26 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 15:58:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 15:58:31 Rebalance progress: 25
2021/12/01 15:58:36 Rebalance progress: 25
2021/12/01 15:58:41 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:58:41 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 15:58:41 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 15:58:41 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9302/api/v1/stats: dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 15:58:41 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 15:58:41 Waiting for src bucket item count to get to: 200000 curr count: 79802
2021/12/01 15:58:46 Waiting for src bucket item count to get to: 200000 curr count: 97181
2021/12/01 15:58:51 Waiting for src bucket item count to get to: 200000 curr count: 97181
2021/12/01 15:58:56 Waiting for src bucket item count to get to: 200000 curr count: 114449
2021/12/01 15:59:01 Waiting for src bucket item count to get to: 200000 curr count: 114449
2021/12/01 15:59:06 Waiting for src bucket item count to get to: 200000 curr count: 131876
2021/12/01 15:59:11 Waiting for src bucket item count to get to: 200000 curr count: 131876
2021/12/01 15:59:16 Waiting for src bucket item count to get to: 200000 curr count: 149175
2021/12/01 15:59:21 Waiting for src bucket item count to get to: 200000 curr count: 149175
2021/12/01 15:59:26 Waiting for src bucket item count to get to: 200000 curr count: 166470
2021/12/01 15:59:31 Waiting for src bucket item count to get to: 200000 curr count: 166470
2021/12/01 15:59:36 Waiting for src bucket item count to get to: 200000 curr count: 183733
2021/12/01 15:59:41 Waiting for src bucket item count to get to: 200000 curr count: 183733
2021/12/01 15:59:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 15:59:56 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:01 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:06 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:11 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:16 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:21 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:26 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:31 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:36 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/01 16:00:36 Post SET ops. Expected item count: 200000 got 200001
2021/12/01 16:00:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:00:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 16:00:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 16:00:52 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 16:00:57 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/01 16:00:57 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/01 16:00:57 No eventing-con process running
2021/12/01 16:00:57 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/01 16:00:58 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:01:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:01:08 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:01:10 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:01:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:01:20 Waiting for dst bucket item count to get to: 0 curr count: 40496
2021/12/01 16:01:25 Waiting for dst bucket item count to get to: 0 curr count: 40496
2021/12/01 16:01:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (273.47s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/01 16:01:30 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:01:30 Removing node(s): [] from the cluster
2021/12/01 16:01:35 Rebalance progress: 100
2021/12/01 16:01:40 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 16:01:40 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/01 16:01:40 No eventing-con process running
2021/12/01 16:01:40 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/01 16:01:42 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:01:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:01:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:01:49 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:01:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:01:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:01:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:01:59 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:02:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 16:02:05 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 16:02:10 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 16:02:15 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 16:02:20 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/01 16:02:20 App: TestDuplicateWithEventingRebOut got deployed
2021/12/01 16:02:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:02:20 Starting bucket ops to source bucket
2021/12/01 16:02:25 Rebalance progress: 25
2021/12/01 16:02:30 Rebalance progress: 25
2021/12/01 16:02:35 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:02:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:02:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:02:35 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9302/api/v1/stats: dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 16:02:35 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:02:35 Waiting for src bucket item count to get to: 200000 curr count: 15242
2021/12/01 16:02:40 Waiting for src bucket item count to get to: 200000 curr count: 32630
2021/12/01 16:02:45 Waiting for src bucket item count to get to: 200000 curr count: 32630
2021/12/01 16:02:50 Waiting for src bucket item count to get to: 200000 curr count: 50121
2021/12/01 16:02:55 Waiting for src bucket item count to get to: 200000 curr count: 50121
2021/12/01 16:03:00 Waiting for src bucket item count to get to: 200000 curr count: 67673
2021/12/01 16:03:05 Waiting for src bucket item count to get to: 200000 curr count: 67673
2021/12/01 16:03:10 Waiting for src bucket item count to get to: 200000 curr count: 67673
2021/12/01 16:03:15 Waiting for src bucket item count to get to: 200000 curr count: 85007
2021/12/01 16:03:20 Waiting for src bucket item count to get to: 200000 curr count: 85007
2021/12/01 16:03:25 Waiting for src bucket item count to get to: 200000 curr count: 102349
2021/12/01 16:03:30 Waiting for src bucket item count to get to: 200000 curr count: 102349
2021/12/01 16:03:35 Waiting for src bucket item count to get to: 200000 curr count: 119904
2021/12/01 16:03:40 Waiting for src bucket item count to get to: 200000 curr count: 119904
2021/12/01 16:03:45 Waiting for src bucket item count to get to: 200000 curr count: 137385
2021/12/01 16:03:50 Waiting for src bucket item count to get to: 200000 curr count: 137385
2021/12/01 16:03:55 Waiting for src bucket item count to get to: 200000 curr count: 154913
2021/12/01 16:04:00 Waiting for src bucket item count to get to: 200000 curr count: 154913
2021/12/01 16:04:05 Waiting for src bucket item count to get to: 200000 curr count: 172444
2021/12/01 16:04:10 Waiting for src bucket item count to get to: 200000 curr count: 172444
2021/12/01 16:04:15 Waiting for src bucket item count to get to: 200000 curr count: 189801
2021/12/01 16:04:20 Waiting for src bucket item count to get to: 200000 curr count: 189801
2021/12/01 16:04:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 16:04:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 16:04:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:04:30 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 16:04:36 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 16:04:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 16:04:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/01 16:04:46 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/01 16:04:46 No eventing-con process running
2021/12/01 16:04:46 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/01 16:04:47 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:04:52 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:04:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:04:59 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:05:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:05:09 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (223.99s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/01 16:05:19 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 16:05:19 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/01 16:05:19 No eventing-con process running
2021/12/01 16:05:19 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/01 16:05:21 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:05:26 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:05:28 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:05:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:05:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:05:38 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:05:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 16:05:44 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 16:05:49 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 16:05:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 16:05:59 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/01 16:05:59 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/01 16:05:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:05:59 Starting bucket ops to source bucket
2021/12/01 16:05:59 Removing node(s): [] from the cluster
2021/12/01 16:06:05 Rebalance progress: 25
2021/12/01 16:06:10 Rebalance progress: 25
2021/12/01 16:06:15 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:06:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:06:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:06:15 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9302/api/v1/stats: dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/01 16:06:15 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:06:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:06:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:06:27 Rebalance progress: 16.66666666666667
2021/12/01 16:06:32 Rebalance progress: 16.66666666666667
2021/12/01 16:06:37 Rebalance progress: 16.66666666666667
2021/12/01 16:06:42 Rebalance progress: 72.22222222222223
2021/12/01 16:06:47 Rebalance progress: 72.22222222222223
2021/12/01 16:06:52 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:06:52 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:06:52 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:06:52 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:06:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:06:57 Rebalance progress: 16.66666666666667
2021/12/01 16:07:02 Rebalance progress: 16.66666666666667
2021/12/01 16:07:07 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:07:07 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:07:07 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:07:07 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:07:07 Waiting for src bucket item count to get to: 200000 curr count: 98276
2021/12/01 16:07:12 Waiting for src bucket item count to get to: 200000 curr count: 98276
2021/12/01 16:07:17 Waiting for src bucket item count to get to: 200000 curr count: 115393
2021/12/01 16:07:22 Waiting for src bucket item count to get to: 200000 curr count: 132586
2021/12/01 16:07:27 Waiting for src bucket item count to get to: 200000 curr count: 132586
2021/12/01 16:07:32 Waiting for src bucket item count to get to: 200000 curr count: 149844
2021/12/01 16:07:37 Waiting for src bucket item count to get to: 200000 curr count: 149844
2021/12/01 16:07:42 Waiting for src bucket item count to get to: 200000 curr count: 167114
2021/12/01 16:07:47 Waiting for src bucket item count to get to: 200000 curr count: 167114
2021/12/01 16:07:52 Waiting for src bucket item count to get to: 200000 curr count: 183943
2021/12/01 16:07:57 Waiting for src bucket item count to get to: 200000 curr count: 183943
2021/12/01 16:08:02 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 16:08:02 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 16:08:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:08:07 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 16:08:12 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 16:08:17 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 16:08:22 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/01 16:08:22 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/01 16:08:22 No eventing-con process running
2021/12/01 16:08:22 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/01 16:08:24 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:08:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:08:34 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:08:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:08:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:08:41 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:08:46 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:08:51 Waiting for dst bucket item count to get to: 0 curr count: 132560
2021/12/01 16:08:56 Waiting for dst bucket item count to get to: 0 curr count: 132560
2021/12/01 16:08:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (226.50s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/01 16:09:06 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/01 16:09:06 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/01 16:09:06 No eventing-con process running
2021/12/01 16:09:06 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/01 16:09:07 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:09:12 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:09:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:09:14 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:09:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:09:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:09:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:09:24 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:09:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 16:09:30 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/01 16:09:35 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/01 16:09:40 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/01 16:09:45 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/01 16:09:45 App: TestDuplicateWithKVRebIn got deployed
2021/12/01 16:09:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 16:09:45 Starting bucket ops to source bucket
2021/12/01 16:09:46 Removing node(s): [] from the cluster
2021/12/01 16:09:51 Rebalance progress: 16.66666666666667
2021/12/01 16:09:56 Rebalance progress: 16.66666666666667
2021/12/01 16:10:01 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:10:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:10:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:10:01 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:10:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:10:06 Rebalance progress: 16.66666666666667
2021/12/01 16:10:11 Rebalance progress: 16.66666666666667
2021/12/01 16:10:16 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:10:16 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:10:16 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:10:16 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:10:16 Waiting for src bucket item count to get to: 200000 curr count: 39117
2021/12/01 16:10:21 Waiting for src bucket item count to get to: 200000 curr count: 56161
2021/12/01 16:10:26 Waiting for src bucket item count to get to: 200000 curr count: 56161
2021/12/01 16:10:31 Waiting for src bucket item count to get to: 200000 curr count: 73358
2021/12/01 16:10:36 Waiting for src bucket item count to get to: 200000 curr count: 73358
2021/12/01 16:10:41 Waiting for src bucket item count to get to: 200000 curr count: 90223
2021/12/01 16:10:46 Waiting for src bucket item count to get to: 200000 curr count: 90223
2021/12/01 16:10:51 Waiting for src bucket item count to get to: 200000 curr count: 90223
2021/12/01 16:10:56 Waiting for src bucket item count to get to: 200000 curr count: 107301
2021/12/01 16:11:01 Waiting for src bucket item count to get to: 200000 curr count: 107301
2021/12/01 16:11:06 Waiting for src bucket item count to get to: 200000 curr count: 124573
2021/12/01 16:11:11 Waiting for src bucket item count to get to: 200000 curr count: 124573
2021/12/01 16:11:16 Waiting for src bucket item count to get to: 200000 curr count: 141764
2021/12/01 16:11:21 Waiting for src bucket item count to get to: 200000 curr count: 141764
2021/12/01 16:11:26 Waiting for src bucket item count to get to: 200000 curr count: 158995
2021/12/01 16:11:31 Waiting for src bucket item count to get to: 200000 curr count: 158995
2021/12/01 16:11:36 Waiting for src bucket item count to get to: 200000 curr count: 176339
2021/12/01 16:11:41 Waiting for src bucket item count to get to: 200000 curr count: 176339
2021/12/01 16:11:46 Waiting for src bucket item count to get to: 200000 curr count: 193307
2021/12/01 16:11:51 Waiting for src bucket item count to get to: 200000 curr count: 193307
2021/12/01 16:11:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 16:11:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 16:11:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:12:01 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/01 16:12:06 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/01 16:12:11 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/01 16:12:16 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/01 16:12:16 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/01 16:12:16 No eventing-con process running
2021/12/01 16:12:16 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/01 16:12:18 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:12:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:12:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:12:30 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:12:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (219.06s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/01 16:12:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 16:12:40 Removing node(s): [] from the cluster
2021/12/01 16:12:45 Rebalance progress: 100
2021/12/01 16:12:50 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/01 16:12:50 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/01 16:12:50 No eventing-con process running
2021/12/01 16:12:50 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/01 16:12:52 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:12:57 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:12:59 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:13:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:13:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:13:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:13:09 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:13:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 16:13:15 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/01 16:13:20 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/01 16:13:25 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/01 16:13:30 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/01 16:13:30 App: TestDuplicateWithKVRebOut got deployed
2021/12/01 16:13:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:13:30 Starting bucket ops to source bucket
2021/12/01 16:13:35 Rebalance progress: 16.66666666666667
2021/12/01 16:13:40 Rebalance progress: 16.66666666666667
2021/12/01 16:13:45 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:13:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:13:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:13:45 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:13:45 Waiting for src bucket item count to get to: 200000 curr count: 15555
2021/12/01 16:13:50 Waiting for src bucket item count to get to: 200000 curr count: 15555
2021/12/01 16:13:55 Waiting for src bucket item count to get to: 200000 curr count: 32540
2021/12/01 16:14:00 Waiting for src bucket item count to get to: 200000 curr count: 49665
2021/12/01 16:14:05 Waiting for src bucket item count to get to: 200000 curr count: 49665
2021/12/01 16:14:10 Waiting for src bucket item count to get to: 200000 curr count: 66924
2021/12/01 16:14:15 Waiting for src bucket item count to get to: 200000 curr count: 66924
2021/12/01 16:14:20 Waiting for src bucket item count to get to: 200000 curr count: 66924
2021/12/01 16:14:25 Waiting for src bucket item count to get to: 200000 curr count: 83837
2021/12/01 16:14:30 Waiting for src bucket item count to get to: 200000 curr count: 83837
2021/12/01 16:14:35 Waiting for src bucket item count to get to: 200000 curr count: 100850
2021/12/01 16:14:40 Waiting for src bucket item count to get to: 200000 curr count: 100850
2021/12/01 16:14:45 Waiting for src bucket item count to get to: 200000 curr count: 118083
2021/12/01 16:14:50 Waiting for src bucket item count to get to: 200000 curr count: 118083
2021/12/01 16:14:55 Waiting for src bucket item count to get to: 200000 curr count: 135281
2021/12/01 16:15:00 Waiting for src bucket item count to get to: 200000 curr count: 135281
2021/12/01 16:15:05 Waiting for src bucket item count to get to: 200000 curr count: 152470
2021/12/01 16:15:10 Waiting for src bucket item count to get to: 200000 curr count: 152470
2021/12/01 16:15:15 Waiting for src bucket item count to get to: 200000 curr count: 169779
2021/12/01 16:15:20 Waiting for src bucket item count to get to: 200000 curr count: 169779
2021/12/01 16:15:25 Waiting for src bucket item count to get to: 200000 curr count: 186628
2021/12/01 16:15:30 Waiting for src bucket item count to get to: 200000 curr count: 186628
2021/12/01 16:15:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 16:15:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 16:15:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:15:40 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/01 16:15:45 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/01 16:15:50 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/01 16:15:55 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/01 16:15:55 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/01 16:15:55 No eventing-con process running
2021/12/01 16:15:55 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/01 16:15:57 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:16:02 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:16:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:16:09 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:16:14 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:16:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (224.03s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/01 16:16:29 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/01 16:16:29 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/01 16:16:29 No eventing-con process running
2021/12/01 16:16:29 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/01 16:16:31 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:16:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:16:37 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:16:42 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:16:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:16:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:16:47 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:16:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 16:16:54 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/01 16:16:59 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/01 16:17:04 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/01 16:17:09 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/01 16:17:09 App: TestDuplicateWithKVSwapReb got deployed
2021/12/01 16:17:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 16:17:09 Starting bucket ops to source bucket
2021/12/01 16:17:09 Removing node(s): [] from the cluster
2021/12/01 16:17:14 Rebalance progress: 16.66666666666667
2021/12/01 16:17:19 Rebalance progress: 16.66666666666667
2021/12/01 16:17:24 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:17:24 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:17:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 16:17:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:17:30 Rebalance progress: 16.66666666666667
2021/12/01 16:17:35 Rebalance progress: 16.66666666666667
2021/12/01 16:17:40 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:40 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:40 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:17:40 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:17:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:17:45 Rebalance progress: 16.66666666666667
2021/12/01 16:17:50 Rebalance progress: 16.66666666666667
2021/12/01 16:17:55 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:55 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:17:55 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:17:55 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:17:55 Waiting for src bucket item count to get to: 200000 curr count: 67044
2021/12/01 16:18:00 Waiting for src bucket item count to get to: 200000 curr count: 67044
2021/12/01 16:18:05 Waiting for src bucket item count to get to: 200000 curr count: 83611
2021/12/01 16:18:10 Waiting for src bucket item count to get to: 200000 curr count: 83611
2021/12/01 16:18:15 Waiting for src bucket item count to get to: 200000 curr count: 100610
2021/12/01 16:18:20 Waiting for src bucket item count to get to: 200000 curr count: 100610
2021/12/01 16:18:25 Waiting for src bucket item count to get to: 200000 curr count: 117761
2021/12/01 16:18:30 Waiting for src bucket item count to get to: 200000 curr count: 117761
2021/12/01 16:18:35 Waiting for src bucket item count to get to: 200000 curr count: 135013
2021/12/01 16:18:40 Waiting for src bucket item count to get to: 200000 curr count: 135013
2021/12/01 16:18:45 Waiting for src bucket item count to get to: 200000 curr count: 152221
2021/12/01 16:18:50 Waiting for src bucket item count to get to: 200000 curr count: 169371
2021/12/01 16:18:55 Waiting for src bucket item count to get to: 200000 curr count: 169371
2021/12/01 16:19:00 Waiting for src bucket item count to get to: 200000 curr count: 186098
2021/12/01 16:19:05 Waiting for src bucket item count to get to: 200000 curr count: 186098
2021/12/01 16:19:10 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 16:19:10 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 16:19:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:19:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/01 16:19:20 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/01 16:19:25 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/01 16:19:30 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/01 16:19:30 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/01 16:19:30 No eventing-con process running
2021/12/01 16:19:30 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/01 16:19:32 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:19:37 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:19:42 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:19:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:19:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:19:49 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:19:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:19:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/01 16:19:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (220.05s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/01 16:20:09 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:20:09 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:20:09 No eventing-con process running
2021/12/01 16:20:09 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/01 16:20:11 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:20:16 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:20:17 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:20:22 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:20:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:20:27 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:20:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/01 16:20:33 Starting bucket ops to source bucket
2021/12/01 16:20:38 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:20:43 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:20:49 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:20:49 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:20:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:20:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:20:49 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:20:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:20:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:20:54 Rebalance progress: 16.66666666666667
2021/12/01 16:20:59 Rebalance progress: 16.66666666666667
2021/12/01 16:21:04 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:21:04 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:21:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:21:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:21:09 Rebalance progress: 16.66666666666667
2021/12/01 16:21:14 Rebalance progress: 16.66666666666667
2021/12/01 16:21:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:21:19 eventingSetup.go:610: http call resp: Get http://127.0.0.1:9303/api/v1/stats: dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/01 16:21:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 16:21:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:21:30 Rebalance progress: 12.5
2021/12/01 16:21:35 Rebalance progress: 12.5
2021/12/01 16:21:40 Rebalance progress: 12.5
2021/12/01 16:21:45 Rebalance progress: 70.83333333333334
2021/12/01 16:21:50 Rebalance progress: 70.83333333333334
2021/12/01 16:21:55 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:55 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:21:55 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:21:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:22:00 Rebalance progress: 12.5
2021/12/01 16:22:05 Rebalance progress: 12.5
2021/12/01 16:22:10 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:10 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:10 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:22:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:22:15 Rebalance progress: 12.5
2021/12/01 16:22:20 Rebalance progress: 12.5
2021/12/01 16:22:25 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:25 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:25 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:22:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 16:22:30 Rebalance progress: 12.5
2021/12/01 16:22:35 Rebalance progress: 12.5
2021/12/01 16:22:40 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:40 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:22:40 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:22:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:22:45 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:22:50 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:22:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:23:00 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:23:00 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:23:00 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:23:00 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/01 16:23:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:23:07 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:23:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12626
2021/12/01 16:23:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12626
2021/12/01 16:23:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:23:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:23:24 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:23:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12626
2021/12/01 16:23:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (209.97s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/01 16:23:39 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/01 16:23:39 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/01 16:23:39 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:23:39 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/01 16:23:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:23:46 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:23:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:23:52 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:23:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:23:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:23:57 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:23:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/01 16:24:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:24:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/01 16:24:14 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/01 16:24:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/01 16:24:19 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:24:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:24:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:24:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 16:24:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:24:25 Rebalance progress: 12.5
2021/12/01 16:24:30 Rebalance progress: 12.5
2021/12/01 16:24:35 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:24:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/01 16:24:40 Rebalance progress: 12.5
2021/12/01 16:24:45 Rebalance progress: 12.5
2021/12/01 16:24:50 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:24:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:24:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:24:55 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/01 16:25:00 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/01 16:25:05 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/01 16:25:10 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/01 16:25:10 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/01 16:25:10 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:25:10 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/01 16:25:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:25:17 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:25:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4633
2021/12/01 16:25:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4633
2021/12/01 16:25:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:25:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:25:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:25:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4633
2021/12/01 16:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.02s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/01 16:25:49 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 16:25:49 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 16:25:49 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:25:49 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/01 16:25:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:25:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:25:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:25:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:03 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:26:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:08 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:26:08 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:26:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/01 16:26:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:26:19 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/01 16:26:24 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/01 16:26:29 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/01 16:26:29 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:29 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:29 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:26:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:26:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:26:34 Rebalance progress: 12.5
2021/12/01 16:26:39 Rebalance progress: 12.5
2021/12/01 16:26:44 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:26:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:26:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:26:49 Rebalance progress: 12.5
2021/12/01 16:26:54 Rebalance progress: 12.5
2021/12/01 16:26:59 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:26:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:26:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 16:27:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:27:05 Rebalance progress: 12.5
2021/12/01 16:27:10 Rebalance progress: 12.5
2021/12/01 16:27:15 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:27:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:27:20 Rebalance progress: 12.5
2021/12/01 16:27:25 Rebalance progress: 12.5
2021/12/01 16:27:30 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:27:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:27:35 Rebalance progress: 12.5
2021/12/01 16:27:40 Rebalance progress: 12.5
2021/12/01 16:27:45 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:27:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:27:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 16:27:50 Rebalance progress: 12.5
2021/12/01 16:27:55 Rebalance progress: 12.5
2021/12/01 16:28:00 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:28:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:28:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:28:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:28:05 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 16:28:10 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 16:28:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 16:28:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/01 16:28:20 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/01 16:28:20 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:28:20 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/01 16:28:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:28:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:28:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10619
2021/12/01 16:28:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10619
2021/12/01 16:28:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:28:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:28:44 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:28:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10614
2021/12/01 16:28:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.07s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/01 16:28:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/01 16:28:59 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/01 16:28:59 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:28:59 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/01 16:29:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:06 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:29:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:29:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:29:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:29:13 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:29:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 16:29:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:29:24 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/01 16:29:29 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/01 16:29:34 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/01 16:29:34 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:29:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:29:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:29:39 Rebalance progress: 12.5
2021/12/01 16:29:44 Rebalance progress: 12.5
2021/12/01 16:29:49 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:29:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:29:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:29:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:29:55 Rebalance progress: 12.5
2021/12/01 16:30:00 Rebalance progress: 12.5
2021/12/01 16:30:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:30:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:30:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:30:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:30:10 Rebalance progress: 12.5
2021/12/01 16:30:15 Rebalance progress: 12.5
2021/12/01 16:30:20 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:30:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:30:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:30:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:30:25 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/01 16:30:30 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/01 16:30:35 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/01 16:30:40 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/01 16:30:40 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/01 16:30:40 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:30:40 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/01 16:30:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:30:47 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:30:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/01 16:30:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/01 16:30:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:30:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:03 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:31:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/01 16:31:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.67s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/01 16:31:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:31:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:31:19 Rebalance progress: 100
2021/12/01 16:31:24 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/01 16:31:24 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/01 16:31:24 No eventing-con process running
2021/12/01 16:31:24 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/01 16:31:26 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:31 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:31:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:31:32 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:37 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:31:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:37 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:31:37 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:31:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/01 16:31:44 Starting bucket ops to source bucket
2021/12/01 16:31:49 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/01 16:31:54 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/01 16:31:59 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/01 16:31:59 App: TestMetaRollbackWithEventingReb got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:31:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:31:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:31:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:32:04 Rebalance progress: 12.5
2021/12/01 16:32:09 Rebalance progress: 12.5
2021/12/01 16:32:14 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:32:14 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:32:14 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:32:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/01 16:32:14 Metakv lookup failed, err: Get /_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0: Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/01 16:32:19 Rebalance progress: 12.5
2021/12/01 16:32:24 Rebalance progress: 12.5
2021/12/01 16:32:29 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:32:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:32:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:32:30 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:32:35 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/01 16:32:40 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/01 16:32:45 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/01 16:32:50 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/01 16:32:50 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/01 16:32:50 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:32:50 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/01 16:32:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:32:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:33:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4571
2021/12/01 16:33:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4571
2021/12/01 16:33:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:33:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:33:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:33:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4571
2021/12/01 16:33:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.72s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/01 16:33:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:33:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:33:29 Rebalance progress: 100
2021/12/01 16:33:34 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/01 16:33:34 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/01 16:33:34 No eventing-con process running
2021/12/01 16:33:34 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/01 16:33:36 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:33:41 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:33:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:33:42 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:33:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:33:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:33:47 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:33:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/01 16:33:53 Starting bucket ops to source bucket
2021/12/01 16:33:58 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/01 16:34:03 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/01 16:34:08 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/01 16:34:09 App: TestMetaPartialRollbackWithEventingReb got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:34:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:34:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:34:14 Rebalance progress: 12.5
2021/12/01 16:34:19 Rebalance progress: 12.5
2021/12/01 16:34:24 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:34:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/01 16:34:24 bucketOps.go:65: Metakv lookup failed, err: Get /_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0: Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/01 16:34:29 Rebalance progress: 12.5
2021/12/01 16:34:34 Rebalance progress: 12.5
2021/12/01 16:34:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:34:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:34:39 bucketOps.go:65: Metakv lookup failed, err: Get /_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0: Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/01 16:34:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:34:45 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/01 16:34:50 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/01 16:34:55 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/01 16:35:00 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/01 16:35:00 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/01 16:35:00 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:35:00 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/01 16:35:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:06 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 43261
2021/12/01 16:35:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 43261
2021/12/01 16:35:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:35:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 43261
2021/12/01 16:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (130.02s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/01 16:35:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/01 16:35:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/01 16:35:38 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:35:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/01 16:35:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:45 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:35:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:52 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:57 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:35:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:35:58 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:35:58 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:35:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 16:36:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:36:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/01 16:36:14 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/01 16:36:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/01 16:36:19 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/01 16:36:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/01 16:36:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/01 16:36:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/01 16:36:34 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:36:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:36:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:36:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:36:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:36:40 Rebalance progress: 12.5
2021/12/01 16:36:45 Rebalance progress: 12.5
2021/12/01 16:36:50 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:36:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:36:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:36:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:36:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:36:55 Rebalance progress: 12.5
2021/12/01 16:37:00 Rebalance progress: 12.5
2021/12/01 16:37:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:37:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 16:37:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:37:10 Rebalance progress: 12.5
2021/12/01 16:37:15 Rebalance progress: 12.5
2021/12/01 16:37:20 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:37:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:37:26 Rebalance progress: 12.5
2021/12/01 16:37:31 Rebalance progress: 12.5
2021/12/01 16:37:36 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:37:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:37:41 Rebalance progress: 12.5
2021/12/01 16:37:46 Rebalance progress: 12.5
2021/12/01 16:37:51 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:37:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:37:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 16:37:56 Rebalance progress: 12.5
2021/12/01 16:38:01 Rebalance progress: 12.5
2021/12/01 16:38:06 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:38:06 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:38:06 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:38:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:38:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/01 16:38:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/01 16:38:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/01 16:38:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/01 16:38:26 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/01 16:38:26 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:38:26 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/01 16:38:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:38:35 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:38:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:38:42 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:38:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12176
2021/12/01 16:38:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:38:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:38:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/01 16:38:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/01 16:39:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/01 16:39:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/01 16:39:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/01 16:39:07 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:39:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/01 16:39:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:39:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:21 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.30s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/01 16:39:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/01 16:39:31 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/01 16:39:31 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:39:31 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/01 16:39:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:37 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:39:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:44 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:49 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:49 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:39:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:39:50 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:39:50 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:39:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 16:39:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:40:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/01 16:40:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/01 16:40:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/01 16:40:11 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/01 16:40:16 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/01 16:40:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/01 16:40:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/01 16:40:27 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:40:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:40:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:40:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/01 16:40:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:40:33 Rebalance progress: 12.5
2021/12/01 16:40:38 Rebalance progress: 12.5
2021/12/01 16:40:43 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:43 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:43 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:40:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/01 16:40:48 Rebalance progress: 12.5
2021/12/01 16:40:53 Rebalance progress: 12.5
2021/12/01 16:40:58 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:40:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:40:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:41:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/01 16:41:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/01 16:41:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/01 16:41:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/01 16:41:18 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/01 16:41:18 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:41:18 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/01 16:41:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:41:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:41:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2758
2021/12/01 16:41:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2758
2021/12/01 16:41:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:41:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:41:44 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:41:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6119
2021/12/01 16:41:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:41:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:41:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/01 16:41:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/01 16:42:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/01 16:42:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/01 16:42:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/01 16:42:09 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:42:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/01 16:42:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:42:16 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:42:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:42:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.96s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/01 16:42:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/01 16:42:33 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/01 16:42:33 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:42:33 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/01 16:42:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:42:39 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:42:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:42:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:42:46 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:42:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:42:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/01 16:42:51 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/01 16:42:51 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:42:51 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/01 16:42:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:42:58 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:42:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:43:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:05 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:43:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:05 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:43:05 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:43:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:06 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:43:06 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:43:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/01 16:43:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:43:17 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/01 16:43:22 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/01 16:43:27 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/01 16:43:27 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/01 16:43:32 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/01 16:43:37 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/01 16:43:42 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/01 16:43:42 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:42 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:43:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:43:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:43:47 Rebalance progress: 12.5
2021/12/01 16:43:52 Rebalance progress: 12.5
2021/12/01 16:43:57 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:57 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:43:57 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:43:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:43:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:44:03 Rebalance progress: 12.5
2021/12/01 16:44:08 Rebalance progress: 12.5
2021/12/01 16:44:13 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:44:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:44:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:44:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:44:18 Rebalance progress: 12.5
2021/12/01 16:44:23 Rebalance progress: 12.5
2021/12/01 16:44:28 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:44:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:44:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:44:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:44:33 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/01 16:44:39 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/01 16:44:44 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/01 16:44:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/01 16:44:49 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/01 16:44:49 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:44:49 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/01 16:44:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:44:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:45:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5457
2021/12/01 16:45:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5457
2021/12/01 16:45:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:45:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:45:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:45:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7638
2021/12/01 16:45:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:45:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:45:24 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/01 16:45:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/01 16:45:34 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/01 16:45:39 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/01 16:45:39 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/01 16:45:39 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:45:39 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/01 16:45:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:45:46 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:45:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:45:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:45:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:45:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.56s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/01 16:46:03 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:46:03 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:46:03 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:46:03 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/01 16:46:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:46:10 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:46:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:46:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:46:17 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:46:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:46:22 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:46:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/01 16:46:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:46:33 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:46:38 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:46:43 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:46:43 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:46:43 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:46:43 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:46:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:46:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:47:03 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:04 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:04 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:47:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:47:39 Rebalance progress: 12.5
2021/12/01 16:47:44 Rebalance progress: 12.5
2021/12/01 16:47:49 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:47:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:47:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:47:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:47:55 Rebalance progress: 12.5
2021/12/01 16:48:00 Rebalance progress: 12.5
2021/12/01 16:48:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:48:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:48:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:48:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:48:10 Rebalance progress: 12.5
2021/12/01 16:48:15 Rebalance progress: 12.5
2021/12/01 16:48:20 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:48:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:48:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:48:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:48:25 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:48:30 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:48:35 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:48:40 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:48:40 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:48:40 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:48:40 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/01 16:48:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:48:47 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:48:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11200
2021/12/01 16:48:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11200
2021/12/01 16:48:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:48:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:04 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:49:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11200
2021/12/01 16:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.64s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/01 16:49:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:49:19 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:49:19 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:49:19 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/01 16:49:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:26 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:49:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:49:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:32 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:49:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:37 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:49:37 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:49:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/01 16:49:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:49:48 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:49:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:49:59 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/01 16:49:59 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:49:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:49:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:50:19 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:29 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:29 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:50:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:50:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:50:50 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:50:50 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:00 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:00 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:51:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:51:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:51:21 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:21 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:51:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:51:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:51:51 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:02 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:02 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:52:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:52:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:52:23 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:52:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:52:38 Rebalance progress: 12.5
2021/12/01 16:52:43 Rebalance progress: 12.5
2021/12/01 16:52:48 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:48 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:52:48 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:52:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:52:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:52:54 Rebalance progress: 12.5
2021/12/01 16:52:59 Rebalance progress: 12.5
2021/12/01 16:53:04 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:53:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:53:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:53:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:53:09 Rebalance progress: 12.5
2021/12/01 16:53:14 Rebalance progress: 12.5
2021/12/01 16:53:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:53:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:53:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:53:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:53:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:53:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:53:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:53:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/01 16:53:39 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/01 16:53:39 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:53:39 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/01 16:53:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:53:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19233
2021/12/01 16:53:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 19233
2021/12/01 16:53:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:53:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:03 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:54:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21529
2021/12/01 16:54:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (299.06s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/01 16:54:18 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/01 16:54:18 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/01 16:54:18 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:54:18 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/01 16:54:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:25 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:54:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:54:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:31 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:54:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:36 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:54:36 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:54:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 16:54:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:54:48 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/01 16:54:53 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/01 16:54:58 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/01 16:54:58 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:54:58 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:54:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 16:54:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:55:03 Rebalance progress: 12.5
2021/12/01 16:55:08 Rebalance progress: 12.5
2021/12/01 16:55:13 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:55:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:55:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:55:28 Rebalance progress: 12.5
2021/12/01 16:55:33 Rebalance progress: 12.5
2021/12/01 16:55:38 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:55:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 16:55:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:55:44 Rebalance progress: 12.5
2021/12/01 16:55:49 Rebalance progress: 12.5
2021/12/01 16:55:54 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:55:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:55:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:55:59 Rebalance progress: 12.5
2021/12/01 16:56:04 Rebalance progress: 12.5
2021/12/01 16:56:09 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:56:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:56:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:56:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:56:14 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/01 16:56:19 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/01 16:56:24 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/01 16:56:29 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/01 16:56:29 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/01 16:56:29 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:56:29 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/01 16:56:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:56:36 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:56:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8211
2021/12/01 16:56:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8211
2021/12/01 16:56:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:56:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:56:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:56:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8633
2021/12/01 16:56:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.99s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/01 16:57:08 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/01 16:57:08 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/01 16:57:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:57:08 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/01 16:57:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:57:15 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:57:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:57:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:57:21 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:57:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:57:26 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:57:26 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:57:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/01 16:57:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 16:57:37 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/01 16:57:43 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/01 16:57:48 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/01 16:57:48 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:57:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:57:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:57:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/01 16:57:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 16:57:53 Rebalance progress: 12.5
2021/12/01 16:57:58 Rebalance progress: 12.5
2021/12/01 16:58:03 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:58:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/01 16:58:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 16:58:08 Rebalance progress: 12.5
2021/12/01 16:58:13 Rebalance progress: 12.5
2021/12/01 16:58:18 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:58:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 16:58:23 Rebalance progress: 12.5
2021/12/01 16:58:28 Rebalance progress: 12.5
2021/12/01 16:58:33 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:58:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 16:58:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 16:58:39 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/01 16:58:44 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/01 16:58:49 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/01 16:58:54 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/01 16:58:54 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/01 16:58:54 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:58:54 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/01 16:58:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:59:00 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:59:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6078
2021/12/01 16:59:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:59:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:59:12 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:59:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6078
2021/12/01 16:59:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.43s)
=== RUN   TestEventingRebInWithOps
2021/12/01 16:59:27 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/01 16:59:27 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/01 16:59:27 eventingSetup.go:46: No eventing-con process running
2021/12/01 16:59:27 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/01 16:59:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:59:34 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:59:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 16:59:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:59:41 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:59:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 16:59:46 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 16:59:46 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 16:59:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 16:59:52 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/01 16:59:57 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/01 17:00:02 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/01 17:00:07 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/01 17:00:07 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/01 17:00:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:00:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 17:00:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 17:00:13 Rebalance progress: 12.5
2021/12/01 17:00:18 Rebalance progress: 12.5
2021/12/01 17:00:23 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:00:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:00:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:00:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:00:28 Rebalance progress: 12.5
2021/12/01 17:00:33 Rebalance progress: 12.5
2021/12/01 17:00:38 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:00:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:00:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:00:38 Waiting for src bucket item count to get to: 200000 curr count: 35643
2021/12/01 17:00:43 Waiting for src bucket item count to get to: 200000 curr count: 52591
2021/12/01 17:00:48 Waiting for src bucket item count to get to: 200000 curr count: 52591
2021/12/01 17:00:53 Waiting for src bucket item count to get to: 200000 curr count: 69644
2021/12/01 17:00:58 Waiting for src bucket item count to get to: 200000 curr count: 69644
2021/12/01 17:01:03 Waiting for src bucket item count to get to: 200000 curr count: 86130
2021/12/01 17:01:08 Waiting for src bucket item count to get to: 200000 curr count: 86130
2021/12/01 17:01:13 Waiting for src bucket item count to get to: 200000 curr count: 102893
2021/12/01 17:01:18 Waiting for src bucket item count to get to: 200000 curr count: 102893
2021/12/01 17:01:23 Waiting for src bucket item count to get to: 200000 curr count: 119803
2021/12/01 17:01:28 Waiting for src bucket item count to get to: 200000 curr count: 119803
2021/12/01 17:01:33 Waiting for src bucket item count to get to: 200000 curr count: 136682
2021/12/01 17:01:38 Waiting for src bucket item count to get to: 200000 curr count: 136682
2021/12/01 17:01:43 Waiting for src bucket item count to get to: 200000 curr count: 153948
2021/12/01 17:01:48 Waiting for src bucket item count to get to: 200000 curr count: 153948
2021/12/01 17:01:53 Waiting for src bucket item count to get to: 200000 curr count: 171135
2021/12/01 17:01:58 Waiting for src bucket item count to get to: 200000 curr count: 171135
2021/12/01 17:02:03 Waiting for src bucket item count to get to: 200000 curr count: 171135
2021/12/01 17:02:08 Waiting for src bucket item count to get to: 200000 curr count: 187541
2021/12/01 17:02:13 Waiting for src bucket item count to get to: 200000 curr count: 187541
2021/12/01 17:02:18 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:02:18 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:02:18 Starting bucket ops to source bucket
2021/12/01 17:06:07 Waiting for dst bucket item count to get to: 0 curr count: 21943
2021/12/01 17:06:12 Waiting for dst bucket item count to get to: 0 curr count: 4143
2021/12/01 17:06:17 Waiting for dst bucket item count to get to: 0 curr count: 4143
2021/12/01 17:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:06:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:06:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:06:22 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/01 17:06:27 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/01 17:06:32 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/01 17:06:37 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/01 17:06:37 App: TestEventingRebInWithOps got un-deployed
2021/12/01 17:06:37 No eventing-con process running
2021/12/01 17:06:37 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/01 17:06:39 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:06:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:06:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:06:45 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:06:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:06:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (453.23s)
=== RUN   TestEventingRebOutWithOps
2021/12/01 17:06:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:06:56 Removing node(s): [] from the cluster
2021/12/01 17:07:01 Rebalance progress: 100
2021/12/01 17:07:06 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/01 17:07:06 App: TestEventingRebOutWithOps got un-deployed
2021/12/01 17:07:06 No eventing-con process running
2021/12/01 17:07:06 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/01 17:07:08 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:07:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:07:14 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:07:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:07:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:07:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:07:24 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:07:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 17:07:31 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/01 17:07:36 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/01 17:07:41 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/01 17:07:46 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/01 17:07:46 App: TestEventingRebOutWithOps got deployed
2021/12/01 17:07:46 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:07:46 Starting bucket ops to source bucket
2021/12/01 17:07:51 Rebalance progress: 12.5
2021/12/01 17:07:56 Rebalance progress: 12.5
2021/12/01 17:08:01 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:08:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:08:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:08:01 Waiting for src bucket item count to get to: 200000 curr count: 22540
2021/12/01 17:08:06 Waiting for src bucket item count to get to: 200000 curr count: 22540
2021/12/01 17:08:11 Waiting for src bucket item count to get to: 200000 curr count: 39610
2021/12/01 17:08:16 Waiting for src bucket item count to get to: 200000 curr count: 39610
2021/12/01 17:08:21 Waiting for src bucket item count to get to: 200000 curr count: 56957
2021/12/01 17:08:26 Waiting for src bucket item count to get to: 200000 curr count: 56957
2021/12/01 17:08:31 Waiting for src bucket item count to get to: 200000 curr count: 74167
2021/12/01 17:08:36 Waiting for src bucket item count to get to: 200000 curr count: 74167
2021/12/01 17:08:41 Waiting for src bucket item count to get to: 200000 curr count: 74167
2021/12/01 17:08:46 Waiting for src bucket item count to get to: 200000 curr count: 90851
2021/12/01 17:08:51 Waiting for src bucket item count to get to: 200000 curr count: 90851
2021/12/01 17:08:56 Waiting for src bucket item count to get to: 200000 curr count: 108027
2021/12/01 17:09:01 Waiting for src bucket item count to get to: 200000 curr count: 108027
2021/12/01 17:09:06 Waiting for src bucket item count to get to: 200000 curr count: 125384
2021/12/01 17:09:11 Waiting for src bucket item count to get to: 200000 curr count: 125384
2021/12/01 17:09:16 Waiting for src bucket item count to get to: 200000 curr count: 142688
2021/12/01 17:09:21 Waiting for src bucket item count to get to: 200000 curr count: 142688
2021/12/01 17:09:26 Waiting for src bucket item count to get to: 200000 curr count: 160119
2021/12/01 17:09:31 Waiting for src bucket item count to get to: 200000 curr count: 160119
2021/12/01 17:09:36 Waiting for src bucket item count to get to: 200000 curr count: 177370
2021/12/01 17:09:41 Waiting for src bucket item count to get to: 200000 curr count: 177370
2021/12/01 17:09:46 Waiting for src bucket item count to get to: 200000 curr count: 194223
2021/12/01 17:09:51 Waiting for src bucket item count to get to: 200000 curr count: 194223
2021/12/01 17:09:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:09:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:09:56 Starting bucket ops to source bucket
2021/12/01 17:13:44 Waiting for dst bucket item count to get to: 0 curr count: 17236
2021/12/01 17:13:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:13:44 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:13:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:13:49 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/01 17:13:54 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/01 17:13:59 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/01 17:14:04 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/01 17:14:04 App: TestEventingRebOutWithOps got un-deployed
2021/12/01 17:14:04 No eventing-con process running
2021/12/01 17:14:04 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/01 17:14:06 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:14:11 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:14:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:14:13 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:14:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:14:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (447.39s)
=== RUN   TestEventingSwapRebWithOps
2021/12/01 17:14:28 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/01 17:14:28 App: TestEventingSwapRebWithOps got un-deployed
2021/12/01 17:14:28 No eventing-con process running
2021/12/01 17:14:28 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/01 17:14:30 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:14:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:14:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:14:36 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:14:41 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:14:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:14:46 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:14:46 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:14:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 17:14:53 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/01 17:14:58 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/01 17:15:03 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/01 17:15:08 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/01 17:15:08 App: TestEventingSwapRebWithOps got deployed
2021/12/01 17:15:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:15:08 Starting bucket ops to source bucket
2021/12/01 17:15:08 Removing node(s): [] from the cluster
2021/12/01 17:15:13 Rebalance progress: 12.5
2021/12/01 17:15:18 Rebalance progress: 12.5
2021/12/01 17:15:23 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:15:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 17:15:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:15:29 Rebalance progress: 12.5
2021/12/01 17:15:34 Rebalance progress: 12.5
2021/12/01 17:15:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:15:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 17:15:44 Rebalance progress: 12.5
2021/12/01 17:15:49 Rebalance progress: 12.5
2021/12/01 17:15:54 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:15:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:15:54 Waiting for src bucket item count to get to: 200000 curr count: 67967
2021/12/01 17:15:59 Waiting for src bucket item count to get to: 200000 curr count: 67967
2021/12/01 17:16:04 Waiting for src bucket item count to get to: 200000 curr count: 84303
2021/12/01 17:16:09 Waiting for src bucket item count to get to: 200000 curr count: 84303
2021/12/01 17:16:14 Waiting for src bucket item count to get to: 200000 curr count: 101234
2021/12/01 17:16:19 Waiting for src bucket item count to get to: 200000 curr count: 101234
2021/12/01 17:16:24 Waiting for src bucket item count to get to: 200000 curr count: 118453
2021/12/01 17:16:29 Waiting for src bucket item count to get to: 200000 curr count: 118453
2021/12/01 17:16:34 Waiting for src bucket item count to get to: 200000 curr count: 135773
2021/12/01 17:16:39 Waiting for src bucket item count to get to: 200000 curr count: 135773
2021/12/01 17:16:44 Waiting for src bucket item count to get to: 200000 curr count: 153022
2021/12/01 17:16:49 Waiting for src bucket item count to get to: 200000 curr count: 153022
2021/12/01 17:16:54 Waiting for src bucket item count to get to: 200000 curr count: 170199
2021/12/01 17:16:59 Waiting for src bucket item count to get to: 200000 curr count: 170199
2021/12/01 17:17:04 Waiting for src bucket item count to get to: 200000 curr count: 170199
2021/12/01 17:17:09 Waiting for src bucket item count to get to: 200000 curr count: 186773
2021/12/01 17:17:14 Waiting for src bucket item count to get to: 200000 curr count: 186773
2021/12/01 17:17:19 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:17:19 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:17:19 Starting bucket ops to source bucket
2021/12/01 17:21:07 Waiting for dst bucket item count to get to: 0 curr count: 22866
2021/12/01 17:21:12 Waiting for dst bucket item count to get to: 0 curr count: 5086
2021/12/01 17:21:17 Waiting for dst bucket item count to get to: 0 curr count: 5086
2021/12/01 17:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:21:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:21:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:21:22 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/01 17:21:27 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/01 17:21:32 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/01 17:21:37 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/01 17:21:37 App: TestEventingSwapRebWithOps got un-deployed
2021/12/01 17:21:37 No eventing-con process running
2021/12/01 17:21:37 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/01 17:21:39 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:21:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:21:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:21:46 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:21:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:21:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (453.08s)
=== RUN   TestKVRebInWithOps
2021/12/01 17:22:01 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/01 17:22:01 App: TestKVRebInWithOps got un-deployed
2021/12/01 17:22:01 No eventing-con process running
2021/12/01 17:22:01 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/01 17:22:03 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:22:08 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:22:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:22:10 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:22:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:22:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:22:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:22:20 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:22:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 17:22:26 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/01 17:22:31 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/01 17:22:36 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/01 17:22:41 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/01 17:22:41 App: TestKVRebInWithOps got deployed
2021/12/01 17:22:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 17:22:41 Starting bucket ops to source bucket
2021/12/01 17:22:41 Removing node(s): [] from the cluster
2021/12/01 17:22:46 Rebalance progress: 12.5
2021/12/01 17:22:51 Rebalance progress: 12.5
2021/12/01 17:22:56 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:22:56 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:22:56 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:22:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:23:01 Rebalance progress: 12.5
2021/12/01 17:23:06 Rebalance progress: 12.5
2021/12/01 17:23:11 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:23:11 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:23:11 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:23:11 Waiting for src bucket item count to get to: 200000 curr count: 46794
2021/12/01 17:23:16 Waiting for src bucket item count to get to: 200000 curr count: 46794
2021/12/01 17:23:21 Waiting for src bucket item count to get to: 200000 curr count: 63683
2021/12/01 17:23:26 Waiting for src bucket item count to get to: 200000 curr count: 63683
2021/12/01 17:23:31 Waiting for src bucket item count to get to: 200000 curr count: 80345
2021/12/01 17:23:36 Waiting for src bucket item count to get to: 200000 curr count: 80345
2021/12/01 17:23:41 Waiting for src bucket item count to get to: 200000 curr count: 97267
2021/12/01 17:23:46 Waiting for src bucket item count to get to: 200000 curr count: 97267
2021/12/01 17:23:51 Waiting for src bucket item count to get to: 200000 curr count: 97267
2021/12/01 17:23:56 Waiting for src bucket item count to get to: 200000 curr count: 114501
2021/12/01 17:24:01 Waiting for src bucket item count to get to: 200000 curr count: 114501
2021/12/01 17:24:06 Waiting for src bucket item count to get to: 200000 curr count: 131765
2021/12/01 17:24:11 Waiting for src bucket item count to get to: 200000 curr count: 131765
2021/12/01 17:24:16 Waiting for src bucket item count to get to: 200000 curr count: 148903
2021/12/01 17:24:21 Waiting for src bucket item count to get to: 200000 curr count: 148903
2021/12/01 17:24:26 Waiting for src bucket item count to get to: 200000 curr count: 166251
2021/12/01 17:24:31 Waiting for src bucket item count to get to: 200000 curr count: 166251
2021/12/01 17:24:36 Waiting for src bucket item count to get to: 200000 curr count: 182849
2021/12/01 17:24:41 Waiting for src bucket item count to get to: 200000 curr count: 182849
2021/12/01 17:24:46 Waiting for src bucket item count to get to: 200000 curr count: 199857
2021/12/01 17:24:51 Waiting for src bucket item count to get to: 200000 curr count: 199857
2021/12/01 17:24:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:24:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:24:56 Starting bucket ops to source bucket
2021/12/01 17:28:45 Waiting for dst bucket item count to get to: 0 curr count: 18668
2021/12/01 17:28:50 Waiting for dst bucket item count to get to: 0 curr count: 351
2021/12/01 17:28:55 Waiting for dst bucket item count to get to: 0 curr count: 351
2021/12/01 17:28:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:28:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:28:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:29:00 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/01 17:29:05 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/01 17:29:10 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/01 17:29:15 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/01 17:29:15 App: TestKVRebInWithOps got un-deployed
2021/12/01 17:29:15 No eventing-con process running
2021/12/01 17:29:15 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/01 17:29:16 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:29:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:29:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:29:23 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:29:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:29:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (457.44s)
=== RUN   TestKVRebOutWithOps
2021/12/01 17:29:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 17:29:34 Removing node(s): [] from the cluster
2021/12/01 17:29:39 Rebalance progress: 100
2021/12/01 17:29:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/01 17:29:44 App: TestKVRebOutWithOps got un-deployed
2021/12/01 17:29:44 No eventing-con process running
2021/12/01 17:29:44 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/01 17:29:45 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:29:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:29:52 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:29:57 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:29:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:30:02 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:30:02 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:30:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 17:30:08 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/01 17:30:13 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/01 17:30:18 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/01 17:30:23 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/01 17:30:23 App: TestKVRebOutWithOps got deployed
2021/12/01 17:30:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:30:23 Starting bucket ops to source bucket
2021/12/01 17:30:28 Rebalance progress: 12.5
2021/12/01 17:30:33 Rebalance progress: 12.5
2021/12/01 17:30:38 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:30:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:30:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:30:39 Waiting for src bucket item count to get to: 200000 curr count: 8841
2021/12/01 17:30:44 Waiting for src bucket item count to get to: 200000 curr count: 25671
2021/12/01 17:30:49 Waiting for src bucket item count to get to: 200000 curr count: 25671
2021/12/01 17:30:54 Waiting for src bucket item count to get to: 200000 curr count: 42818
2021/12/01 17:30:59 Waiting for src bucket item count to get to: 200000 curr count: 42818
2021/12/01 17:31:04 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/01 17:31:09 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/01 17:31:14 Waiting for src bucket item count to get to: 200000 curr count: 59935
2021/12/01 17:31:19 Waiting for src bucket item count to get to: 200000 curr count: 76790
2021/12/01 17:31:24 Waiting for src bucket item count to get to: 200000 curr count: 76790
2021/12/01 17:31:29 Waiting for src bucket item count to get to: 200000 curr count: 93578
2021/12/01 17:31:34 Waiting for src bucket item count to get to: 200000 curr count: 93578
2021/12/01 17:31:39 Waiting for src bucket item count to get to: 200000 curr count: 110454
2021/12/01 17:31:44 Waiting for src bucket item count to get to: 200000 curr count: 110454
2021/12/01 17:31:49 Waiting for src bucket item count to get to: 200000 curr count: 127684
2021/12/01 17:31:54 Waiting for src bucket item count to get to: 200000 curr count: 127684
2021/12/01 17:31:59 Waiting for src bucket item count to get to: 200000 curr count: 144897
2021/12/01 17:32:04 Waiting for src bucket item count to get to: 200000 curr count: 144897
2021/12/01 17:32:09 Waiting for src bucket item count to get to: 200000 curr count: 161974
2021/12/01 17:32:14 Waiting for src bucket item count to get to: 200000 curr count: 161974
2021/12/01 17:32:19 Waiting for src bucket item count to get to: 200000 curr count: 179126
2021/12/01 17:32:24 Waiting for src bucket item count to get to: 200000 curr count: 195875
2021/12/01 17:32:29 Waiting for src bucket item count to get to: 200000 curr count: 195875
2021/12/01 17:32:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:32:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:32:34 Starting bucket ops to source bucket
2021/12/01 17:36:22 Waiting for dst bucket item count to get to: 0 curr count: 14179
2021/12/01 17:36:27 Waiting for dst bucket item count to get to: 0 curr count: 14179
2021/12/01 17:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:36:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:36:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:36:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/01 17:36:37 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/01 17:36:42 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/01 17:36:47 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/01 17:36:47 App: TestKVRebOutWithOps got un-deployed
2021/12/01 17:36:47 No eventing-con process running
2021/12/01 17:36:47 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/01 17:36:49 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:36:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:36:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:36:56 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:37:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:37:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (452.73s)
=== RUN   TestKVSwapRebWithOps
2021/12/01 17:37:11 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/01 17:37:11 App: TestKVSwapRebWithOps got un-deployed
2021/12/01 17:37:11 No eventing-con process running
2021/12/01 17:37:11 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/01 17:37:13 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:37:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:37:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:37:20 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:37:25 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:37:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:37:30 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:37:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/01 17:37:36 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/01 17:37:41 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/01 17:37:46 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/01 17:37:51 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/01 17:37:51 App: TestKVSwapRebWithOps got deployed
2021/12/01 17:37:51 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 17:37:51 Starting bucket ops to source bucket
2021/12/01 17:37:52 Removing node(s): [] from the cluster
2021/12/01 17:37:57 Rebalance progress: 12.5
2021/12/01 17:38:02 Rebalance progress: 12.5
2021/12/01 17:38:07 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:07 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:07 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:38:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 17:38:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:38:13 Rebalance progress: 12.5
2021/12/01 17:38:18 Rebalance progress: 12.5
2021/12/01 17:38:23 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:23 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:38:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 17:38:28 Rebalance progress: 12.5
2021/12/01 17:38:33 Rebalance progress: 12.5
2021/12/01 17:38:38 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:38:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:38:38 Waiting for src bucket item count to get to: 200000 curr count: 61787
2021/12/01 17:38:43 Waiting for src bucket item count to get to: 200000 curr count: 78113
2021/12/01 17:38:48 Waiting for src bucket item count to get to: 200000 curr count: 78113
2021/12/01 17:38:53 Waiting for src bucket item count to get to: 200000 curr count: 95027
2021/12/01 17:38:58 Waiting for src bucket item count to get to: 200000 curr count: 95027
2021/12/01 17:39:03 Waiting for src bucket item count to get to: 200000 curr count: 112275
2021/12/01 17:39:08 Waiting for src bucket item count to get to: 200000 curr count: 112275
2021/12/01 17:39:13 Waiting for src bucket item count to get to: 200000 curr count: 129413
2021/12/01 17:39:18 Waiting for src bucket item count to get to: 200000 curr count: 129413
2021/12/01 17:39:23 Waiting for src bucket item count to get to: 200000 curr count: 146605
2021/12/01 17:39:28 Waiting for src bucket item count to get to: 200000 curr count: 146605
2021/12/01 17:39:33 Waiting for src bucket item count to get to: 200000 curr count: 163884
2021/12/01 17:39:38 Waiting for src bucket item count to get to: 200000 curr count: 163884
2021/12/01 17:39:43 Waiting for src bucket item count to get to: 200000 curr count: 180487
2021/12/01 17:39:48 Waiting for src bucket item count to get to: 200000 curr count: 180487
2021/12/01 17:39:53 Waiting for src bucket item count to get to: 200000 curr count: 197507
2021/12/01 17:39:58 Waiting for src bucket item count to get to: 200000 curr count: 197507
2021/12/01 17:40:03 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/01 17:40:03 Post SET ops. Expected item count: 200000 got 200000
2021/12/01 17:40:03 Starting bucket ops to source bucket
2021/12/01 17:43:51 Waiting for dst bucket item count to get to: 0 curr count: 12801
2021/12/01 17:43:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:43:51 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:43:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:43:56 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/01 17:44:01 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/01 17:44:06 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/01 17:44:11 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/01 17:44:11 App: TestKVSwapRebWithOps got un-deployed
2021/12/01 17:44:11 No eventing-con process running
2021/12/01 17:44:11 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/01 17:44:13 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:44:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:44:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:44:20 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:44:25 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (443.95s)
=== RUN   TestEventingRebInWithTimers
2021/12/01 17:44:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/01 17:44:35 App: TestEventingRebInWithTimers got un-deployed
2021/12/01 17:44:35 No eventing-con process running
2021/12/01 17:44:35 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/01 17:44:37 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:44:42 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:44:44 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:44:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:44:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:44:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:44:54 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:44:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 17:45:00 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/01 17:45:05 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/01 17:45:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/01 17:45:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/01 17:45:15 App: TestEventingRebInWithTimers got deployed
2021/12/01 17:45:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:45:15 Starting bucket ops to source bucket
2021/12/01 17:45:15 Removing node(s): [] from the cluster
2021/12/01 17:45:20 Rebalance progress: 12.5
2021/12/01 17:45:25 Rebalance progress: 12.5
2021/12/01 17:45:30 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:45:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:45:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:45:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:45:36 Rebalance progress: 12.5
2021/12/01 17:45:41 Rebalance progress: 12.5
2021/12/01 17:45:46 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:45:46 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:45:46 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:45:46 Waiting for src bucket item count to get to: 50000 curr count: 34683
2021/12/01 17:45:51 Waiting for src bucket item count to get to: 50000 curr count: 48705
2021/12/01 17:45:56 Waiting for src bucket item count to get to: 50000 curr count: 48705
2021/12/01 17:46:06 Waiting for dst bucket item count to get to: 50000 curr count: 47418
2021/12/01 17:46:11 Waiting for dst bucket item count to get to: 50000 curr count: 47418
2021/12/01 17:46:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 17:46:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 17:46:11 Starting bucket ops to source bucket
2021/12/01 17:47:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:47:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:47:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:47:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:47:40 Waiting for dst bucket item count to get to: 0 curr count: 34657
2021/12/01 17:47:45 Waiting for dst bucket item count to get to: 0 curr count: 34657
2021/12/01 17:47:50 Waiting for dst bucket item count to get to: 0 curr count: 19171
2021/12/01 17:47:55 Waiting for dst bucket item count to get to: 0 curr count: 19171
2021/12/01 17:48:00 Waiting for dst bucket item count to get to: 0 curr count: 2427
2021/12/01 17:48:05 Waiting for dst bucket item count to get to: 0 curr count: 2427
2021/12/01 17:48:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:48:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:48:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:48:10 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/01 17:48:15 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/01 17:48:20 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/01 17:48:25 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/01 17:48:25 App: TestEventingRebInWithTimers got un-deployed
2021/12/01 17:48:25 No eventing-con process running
2021/12/01 17:48:25 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/01 17:48:27 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:48:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:48:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:48:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:48:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:48:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (253.58s)
=== RUN   TestEventingRebOutWithTimers
2021/12/01 17:48:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:48:44 Removing node(s): [] from the cluster
2021/12/01 17:48:49 Rebalance progress: 100
2021/12/01 17:48:54 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/01 17:48:54 App: TestEventingRebOutWithTimers got un-deployed
2021/12/01 17:48:54 No eventing-con process running
2021/12/01 17:48:54 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/01 17:48:56 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:49:01 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:49:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:49:03 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:49:08 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:49:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:49:13 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:49:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 17:49:19 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/01 17:49:24 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/01 17:49:29 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/01 17:49:34 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/01 17:49:34 App: TestEventingRebOutWithTimers got deployed
2021/12/01 17:49:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:49:34 Starting bucket ops to source bucket
2021/12/01 17:49:39 Rebalance progress: 12.5
2021/12/01 17:49:44 Rebalance progress: 12.5
2021/12/01 17:49:49 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:49:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:49:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:49:49 Waiting for src bucket item count to get to: 50000 curr count: 8010
2021/12/01 17:49:54 Waiting for src bucket item count to get to: 50000 curr count: 23631
2021/12/01 17:49:59 Waiting for src bucket item count to get to: 50000 curr count: 23631
2021/12/01 17:50:04 Waiting for src bucket item count to get to: 50000 curr count: 37846
2021/12/01 17:50:09 Waiting for src bucket item count to get to: 50000 curr count: 37846
2021/12/01 17:50:19 Waiting for dst bucket item count to get to: 50000 curr count: 34813
2021/12/01 17:50:24 Waiting for dst bucket item count to get to: 50000 curr count: 34813
2021/12/01 17:50:29 Waiting for dst bucket item count to get to: 50000 curr count: 49673
2021/12/01 17:50:34 Waiting for dst bucket item count to get to: 50000 curr count: 49673
2021/12/01 17:50:39 Waiting for dst bucket item count to get to: 50000 curr count: 49673
2021/12/01 17:50:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 17:50:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 17:50:39 Starting bucket ops to source bucket
2021/12/01 17:51:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:51:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:51:58 Waiting for dst bucket item count to get to: 0 curr count: 46350
2021/12/01 17:52:03 Waiting for dst bucket item count to get to: 0 curr count: 46350
2021/12/01 17:52:08 Waiting for dst bucket item count to get to: 0 curr count: 38032
2021/12/01 17:52:13 Waiting for dst bucket item count to get to: 0 curr count: 38032
2021/12/01 17:52:18 Waiting for dst bucket item count to get to: 0 curr count: 13642
2021/12/01 17:52:23 Waiting for dst bucket item count to get to: 0 curr count: 13642
2021/12/01 17:52:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:52:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:52:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:52:28 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/01 17:52:33 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/01 17:52:38 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/01 17:52:43 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/01 17:52:43 App: TestEventingRebOutWithTimers got un-deployed
2021/12/01 17:52:43 No eventing-con process running
2021/12/01 17:52:43 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/01 17:52:45 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:52:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:52:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:52:52 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:52:57 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:52:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (258.05s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/01 17:53:07 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/01 17:53:07 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/01 17:53:07 No eventing-con process running
2021/12/01 17:53:07 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/01 17:53:08 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:53:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:53:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:53:15 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:53:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:53:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:53:25 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:53:25 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:53:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 17:53:31 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/01 17:53:36 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/01 17:53:41 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/01 17:53:46 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/01 17:53:46 App: TestEventingSwapRebWithTimers got deployed
2021/12/01 17:53:46 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 17:53:46 Starting bucket ops to source bucket
2021/12/01 17:53:47 Removing node(s): [] from the cluster
2021/12/01 17:53:52 Rebalance progress: 12.5
2021/12/01 17:53:57 Rebalance progress: 12.5
2021/12/01 17:54:02 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:02 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:02 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:54:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 17:54:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:54:07 Rebalance progress: 12.5
2021/12/01 17:54:12 Rebalance progress: 12.5
2021/12/01 17:54:17 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:17 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:17 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:54:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 17:54:23 Rebalance progress: 12.5
2021/12/01 17:54:28 Rebalance progress: 12.5
2021/12/01 17:54:33 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:54:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:54:38 Waiting for dst bucket item count to get to: 50000 curr count: 42727
2021/12/01 17:54:43 Waiting for dst bucket item count to get to: 50000 curr count: 42727
2021/12/01 17:54:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 17:54:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 17:54:43 Starting bucket ops to source bucket
2021/12/01 17:55:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:55:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:56:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:56:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 17:56:12 Waiting for dst bucket item count to get to: 0 curr count: 43965
2021/12/01 17:56:17 Waiting for dst bucket item count to get to: 0 curr count: 43965
2021/12/01 17:56:22 Waiting for dst bucket item count to get to: 0 curr count: 22654
2021/12/01 17:56:27 Waiting for dst bucket item count to get to: 0 curr count: 22654
2021/12/01 17:56:32 Waiting for dst bucket item count to get to: 0 curr count: 9218
2021/12/01 17:56:37 Waiting for dst bucket item count to get to: 0 curr count: 9218
2021/12/01 17:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:56:37 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 17:56:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 17:56:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/01 17:56:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/01 17:56:52 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/01 17:56:57 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/01 17:56:57 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/01 17:56:57 No eventing-con process running
2021/12/01 17:56:57 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/01 17:56:59 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:57:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:57:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:57:06 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:57:11 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:57:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (254.14s)
=== RUN   TestKVRebInWithTimers
2021/12/01 17:57:21 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/01 17:57:21 App: TestKVRebInWithTimers got un-deployed
2021/12/01 17:57:21 No eventing-con process running
2021/12/01 17:57:21 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/01 17:57:23 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:57:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:57:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 17:57:29 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:57:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:57:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 17:57:39 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 17:57:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 17:57:46 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/01 17:57:51 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/01 17:57:56 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/01 17:58:01 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/01 17:58:01 App: TestKVRebInWithTimers got deployed
2021/12/01 17:58:01 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 17:58:01 Starting bucket ops to source bucket
2021/12/01 17:58:01 Removing node(s): [] from the cluster
2021/12/01 17:58:06 Rebalance progress: 12.5
2021/12/01 17:58:11 Rebalance progress: 12.5
2021/12/01 17:58:16 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:58:16 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:58:16 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:58:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 17:58:21 Rebalance progress: 12.5
2021/12/01 17:58:26 Rebalance progress: 12.5
2021/12/01 17:58:31 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:58:31 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 17:58:31 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 17:58:31 Waiting for src bucket item count to get to: 50000 curr count: 41792
2021/12/01 17:58:36 Waiting for src bucket item count to get to: 50000 curr count: 41792
2021/12/01 17:58:46 Waiting for dst bucket item count to get to: 50000 curr count: 30122
2021/12/01 17:58:51 Waiting for dst bucket item count to get to: 50000 curr count: 30122
2021/12/01 17:58:56 Waiting for dst bucket item count to get to: 50000 curr count: 41410
2021/12/01 17:59:01 Waiting for dst bucket item count to get to: 50000 curr count: 41410
2021/12/01 17:59:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 17:59:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 17:59:01 Starting bucket ops to source bucket
2021/12/01 18:00:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:00:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:00:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:00:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:00:30 Waiting for dst bucket item count to get to: 0 curr count: 42730
2021/12/01 18:00:35 Waiting for dst bucket item count to get to: 0 curr count: 42730
2021/12/01 18:00:40 Waiting for dst bucket item count to get to: 0 curr count: 22142
2021/12/01 18:00:45 Waiting for dst bucket item count to get to: 0 curr count: 22142
2021/12/01 18:00:50 Waiting for dst bucket item count to get to: 0 curr count: 7849
2021/12/01 18:00:55 Waiting for dst bucket item count to get to: 0 curr count: 7849
2021/12/01 18:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:00:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 18:00:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:01:00 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/01 18:01:05 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/01 18:01:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/01 18:01:15 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/01 18:01:15 App: TestKVRebInWithTimers got un-deployed
2021/12/01 18:01:15 No eventing-con process running
2021/12/01 18:01:15 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/01 18:01:17 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:01:22 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:01:24 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:01:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:01:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.14s)
=== RUN   TestKVRebOutWithTimers
2021/12/01 18:01:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:01:34 Removing node(s): [] from the cluster
2021/12/01 18:01:39 Rebalance progress: 100
2021/12/01 18:01:44 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/01 18:01:44 App: TestKVRebOutWithTimers got un-deployed
2021/12/01 18:01:44 No eventing-con process running
2021/12/01 18:01:44 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/01 18:01:46 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:01:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:01:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:01:53 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:01:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:01:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:02:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:02:03 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:02:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 18:02:09 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/01 18:02:14 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/01 18:02:19 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/01 18:02:24 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/01 18:02:24 App: TestKVRebOutWithTimers got deployed
2021/12/01 18:02:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:02:24 Starting bucket ops to source bucket
2021/12/01 18:02:29 Rebalance progress: 12.5
2021/12/01 18:02:34 Rebalance progress: 12.5
2021/12/01 18:02:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:02:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:02:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:02:39 Waiting for src bucket item count to get to: 50000 curr count: 7535
2021/12/01 18:02:44 Waiting for src bucket item count to get to: 50000 curr count: 22957
2021/12/01 18:02:49 Waiting for src bucket item count to get to: 50000 curr count: 22957
2021/12/01 18:02:54 Waiting for src bucket item count to get to: 50000 curr count: 36921
2021/12/01 18:02:59 Waiting for src bucket item count to get to: 50000 curr count: 36921
2021/12/01 18:03:09 Waiting for dst bucket item count to get to: 50000 curr count: 32422
2021/12/01 18:03:14 Waiting for dst bucket item count to get to: 50000 curr count: 32422
2021/12/01 18:03:19 Waiting for dst bucket item count to get to: 50000 curr count: 32422
2021/12/01 18:03:24 Waiting for dst bucket item count to get to: 50000 curr count: 48291
2021/12/01 18:03:29 Waiting for dst bucket item count to get to: 50000 curr count: 48291
2021/12/01 18:03:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:03:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:03:29 Starting bucket ops to source bucket
2021/12/01 18:04:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:04:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:04:49 Waiting for dst bucket item count to get to: 0 curr count: 47885
2021/12/01 18:04:54 Waiting for dst bucket item count to get to: 0 curr count: 47885
2021/12/01 18:04:59 Waiting for dst bucket item count to get to: 0 curr count: 40143
2021/12/01 18:05:04 Waiting for dst bucket item count to get to: 0 curr count: 40143
2021/12/01 18:05:09 Waiting for dst bucket item count to get to: 0 curr count: 16677
2021/12/01 18:05:14 Waiting for dst bucket item count to get to: 0 curr count: 16677
2021/12/01 18:05:19 Waiting for dst bucket item count to get to: 0 curr count: 1136
2021/12/01 18:05:24 Waiting for dst bucket item count to get to: 0 curr count: 1136
2021/12/01 18:05:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:05:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 18:05:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:05:29 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/01 18:05:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/01 18:05:39 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/01 18:05:44 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/01 18:05:44 App: TestKVRebOutWithTimers got un-deployed
2021/12/01 18:05:44 No eventing-con process running
2021/12/01 18:05:44 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/01 18:05:46 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:05:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:05:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:05:53 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:05:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.12s)
=== RUN   TestKVSwapRebWithTimers
2021/12/01 18:06:08 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:06:08 App: TestKVSwapRebWithTimers got un-deployed
2021/12/01 18:06:08 No eventing-con process running
2021/12/01 18:06:08 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/01 18:06:10 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:06:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:06:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:06:17 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:06:22 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:06:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:06:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:06:27 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:06:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/01 18:06:33 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/01 18:06:38 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/01 18:06:43 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/01 18:06:48 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/01 18:06:48 App: TestKVSwapRebWithTimers got deployed
2021/12/01 18:06:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:06:48 Starting bucket ops to source bucket
2021/12/01 18:06:48 Removing node(s): [] from the cluster
2021/12/01 18:06:53 Rebalance progress: 12.5
2021/12/01 18:06:58 Rebalance progress: 12.5
2021/12/01 18:07:03 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:07:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:07:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:07:09 Rebalance progress: 12.5
2021/12/01 18:07:14 Rebalance progress: 12.5
2021/12/01 18:07:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:07:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:07:24 Rebalance progress: 12.5
2021/12/01 18:07:29 Rebalance progress: 12.5
2021/12/01 18:07:34 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:07:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:07:39 Waiting for dst bucket item count to get to: 50000 curr count: 30909
2021/12/01 18:07:44 Waiting for dst bucket item count to get to: 50000 curr count: 30909
2021/12/01 18:07:49 Waiting for dst bucket item count to get to: 50000 curr count: 48032
2021/12/01 18:07:54 Waiting for dst bucket item count to get to: 50000 curr count: 48032
2021/12/01 18:07:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:07:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:07:54 Starting bucket ops to source bucket
2021/12/01 18:09:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:09:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:09:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:09:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:09:23 Waiting for dst bucket item count to get to: 0 curr count: 41642
2021/12/01 18:09:28 Waiting for dst bucket item count to get to: 0 curr count: 31953
2021/12/01 18:09:33 Waiting for dst bucket item count to get to: 0 curr count: 31953
2021/12/01 18:09:38 Waiting for dst bucket item count to get to: 0 curr count: 17048
2021/12/01 18:09:43 Waiting for dst bucket item count to get to: 0 curr count: 17048
2021/12/01 18:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:09:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/01 18:09:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:09:48 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:09:53 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:09:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:10:03 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:10:08 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:10:13 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/01 18:10:13 App: TestKVSwapRebWithTimers got un-deployed
2021/12/01 18:10:13 No eventing-con process running
2021/12/01 18:10:13 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/01 18:10:15 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:10:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:10:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:10:22 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:10:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:10:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (268.98s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/01 18:10:37 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:10:37 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/01 18:10:37 No eventing-con process running
2021/12/01 18:10:37 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/01 18:10:39 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:10:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:10:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:10:46 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:10:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:10:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:10:56 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:10:56 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:10:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 18:11:02 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/01 18:11:07 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/01 18:11:12 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/01 18:11:17 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/01 18:11:17 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/01 18:11:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:11:17 Starting bucket ops to source bucket
2021/12/01 18:11:17 Removing node(s): [] from the cluster
2021/12/01 18:11:22 Rebalance progress: 12.5
2021/12/01 18:11:27 Rebalance progress: 12.5
2021/12/01 18:11:32 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:11:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:11:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:11:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:11:37 Rebalance progress: 12.5
2021/12/01 18:11:42 Rebalance progress: 12.5
2021/12/01 18:11:47 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:11:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:11:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:11:47 Waiting for src bucket item count to get to: 50000 curr count: 31952
2021/12/01 18:11:52 Waiting for src bucket item count to get to: 50000 curr count: 46087
2021/12/01 18:11:57 Waiting for src bucket item count to get to: 50000 curr count: 46087
2021/12/01 18:12:07 Waiting for dst bucket item count to get to: 50000 curr count: 29153
2021/12/01 18:12:12 Waiting for dst bucket item count to get to: 50000 curr count: 29153
2021/12/01 18:12:17 Waiting for dst bucket item count to get to: 50000 curr count: 40091
2021/12/01 18:12:22 Waiting for dst bucket item count to get to: 50000 curr count: 40091
2021/12/01 18:12:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:12:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:12:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:12:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/01 18:12:53 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/01 18:12:53 No eventing-con process running
2021/12/01 18:12:53 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/01 18:12:55 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:13:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:13:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:13:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:13:06 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:13:11 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:13:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (169.41s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/01 18:13:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:13:22 Removing node(s): [] from the cluster
2021/12/01 18:13:27 Rebalance progress: 100
2021/12/01 18:13:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:13:32 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/01 18:13:32 No eventing-con process running
2021/12/01 18:13:32 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/01 18:13:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:13:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:13:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:13:40 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:13:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:13:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:13:50 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:13:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 18:13:56 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/01 18:14:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/01 18:14:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/01 18:14:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/01 18:14:12 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/01 18:14:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:14:12 Starting bucket ops to source bucket
2021/12/01 18:14:17 Rebalance progress: 12.5
2021/12/01 18:14:22 Rebalance progress: 12.5
2021/12/01 18:14:27 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:14:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:14:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:14:27 Waiting for src bucket item count to get to: 50000 curr count: 10981
2021/12/01 18:14:32 Waiting for src bucket item count to get to: 50000 curr count: 26458
2021/12/01 18:14:37 Waiting for src bucket item count to get to: 50000 curr count: 26458
2021/12/01 18:14:42 Waiting for src bucket item count to get to: 50000 curr count: 41155
2021/12/01 18:14:47 Waiting for src bucket item count to get to: 50000 curr count: 41155
2021/12/01 18:14:57 Waiting for dst bucket item count to get to: 50000 curr count: 26739
2021/12/01 18:15:02 Waiting for dst bucket item count to get to: 50000 curr count: 26739
2021/12/01 18:15:07 Waiting for dst bucket item count to get to: 50000 curr count: 26739
2021/12/01 18:15:12 Waiting for dst bucket item count to get to: 50000 curr count: 36894
2021/12/01 18:15:17 Waiting for dst bucket item count to get to: 50000 curr count: 36894
2021/12/01 18:15:17 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:15:17 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:15:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:15:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/01 18:15:47 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/01 18:15:47 No eventing-con process running
2021/12/01 18:15:47 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/01 18:15:49 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:15:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:15:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:15:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:16:01 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:16:06 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:16:11 Waiting for dst bucket item count to get to: 0 curr count: 36482
2021/12/01 18:16:16 Waiting for dst bucket item count to get to: 0 curr count: 36482
2021/12/01 18:16:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.25s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/01 18:16:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:16:26 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/01 18:16:26 No eventing-con process running
2021/12/01 18:16:26 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/01 18:16:28 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:16:33 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:16:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:16:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:16:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:16:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:16:44 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:16:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 18:16:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/01 18:16:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/01 18:17:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/01 18:17:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/01 18:17:06 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/01 18:17:06 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:17:06 Starting bucket ops to source bucket
2021/12/01 18:17:06 Removing node(s): [] from the cluster
2021/12/01 18:17:11 Rebalance progress: 12.5
2021/12/01 18:17:16 Rebalance progress: 12.5
2021/12/01 18:17:21 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:17:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 18:17:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:17:26 Rebalance progress: 12.5
2021/12/01 18:17:31 Rebalance progress: 12.5
2021/12/01 18:17:36 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:17:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:17:41 Rebalance progress: 12.5
2021/12/01 18:17:46 Rebalance progress: 12.5
2021/12/01 18:17:51 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:17:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:17:57 Waiting for dst bucket item count to get to: 50000 curr count: 37159
2021/12/01 18:18:02 Waiting for dst bucket item count to get to: 50000 curr count: 37159
2021/12/01 18:18:07 Waiting for dst bucket item count to get to: 50000 curr count: 42817
2021/12/01 18:18:12 Waiting for dst bucket item count to get to: 50000 curr count: 42817
2021/12/01 18:18:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:18:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:18:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:18:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/01 18:18:42 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/01 18:18:42 No eventing-con process running
2021/12/01 18:18:42 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/01 18:18:44 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:18:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:18:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:18:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:18:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:19:00 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:19:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:19:10 Waiting for dst bucket item count to get to: 0 curr count: 27469
2021/12/01 18:19:16 Waiting for dst bucket item count to get to: 0 curr count: 27469
2021/12/01 18:19:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.84s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/01 18:19:26 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:19:26 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/01 18:19:26 No eventing-con process running
2021/12/01 18:19:26 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/01 18:19:27 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:19:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:19:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:19:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:19:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:19:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:19:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:19:44 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:19:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 18:19:50 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/01 18:19:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/01 18:20:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/01 18:20:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/01 18:20:05 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/01 18:20:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:20:05 Starting bucket ops to source bucket
2021/12/01 18:20:06 Removing node(s): [] from the cluster
2021/12/01 18:20:11 Rebalance progress: 12.5
2021/12/01 18:20:16 Rebalance progress: 12.5
2021/12/01 18:20:21 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:20:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:20:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:20:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:20:26 Rebalance progress: 12.5
2021/12/01 18:20:31 Rebalance progress: 12.5
2021/12/01 18:20:36 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:20:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:20:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:20:36 Waiting for src bucket item count to get to: 50000 curr count: 35494
2021/12/01 18:20:41 Waiting for src bucket item count to get to: 50000 curr count: 49709
2021/12/01 18:20:46 Waiting for src bucket item count to get to: 50000 curr count: 49709
2021/12/01 18:20:56 Waiting for dst bucket item count to get to: 50000 curr count: 30936
2021/12/01 18:21:01 Waiting for dst bucket item count to get to: 50000 curr count: 30936
2021/12/01 18:21:06 Waiting for dst bucket item count to get to: 50000 curr count: 40859
2021/12/01 18:21:11 Waiting for dst bucket item count to get to: 50000 curr count: 40859
2021/12/01 18:21:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:21:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:21:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:21:17 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:22 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:32 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:37 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/01 18:21:42 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/01 18:21:42 No eventing-con process running
2021/12/01 18:21:42 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/01 18:21:44 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:21:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:21:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:21:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:22:00 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:22:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:22:10 Waiting for dst bucket item count to get to: 0 curr count: 21058
2021/12/01 18:22:15 Waiting for dst bucket item count to get to: 0 curr count: 21058
2021/12/01 18:22:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.85s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/01 18:22:20 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:22:21 Removing node(s): [] from the cluster
2021/12/01 18:22:26 Rebalance progress: 100
2021/12/01 18:22:31 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:22:31 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/01 18:22:31 No eventing-con process running
2021/12/01 18:22:31 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/01 18:22:33 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:22:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:22:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:22:40 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:22:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:22:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:22:50 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:22:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 18:22:56 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/01 18:23:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/01 18:23:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/01 18:23:11 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/01 18:23:11 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/01 18:23:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:23:11 Starting bucket ops to source bucket
2021/12/01 18:23:16 Rebalance progress: 12.5
2021/12/01 18:23:21 Rebalance progress: 12.5
2021/12/01 18:23:26 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:23:26 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:23:26 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:23:26 Waiting for src bucket item count to get to: 50000 curr count: 12159
2021/12/01 18:23:31 Waiting for src bucket item count to get to: 50000 curr count: 27735
2021/12/01 18:23:36 Waiting for src bucket item count to get to: 50000 curr count: 27735
2021/12/01 18:23:41 Waiting for src bucket item count to get to: 50000 curr count: 42673
2021/12/01 18:23:46 Waiting for src bucket item count to get to: 50000 curr count: 42673
2021/12/01 18:23:56 Waiting for dst bucket item count to get to: 50000 curr count: 26426
2021/12/01 18:24:01 Waiting for dst bucket item count to get to: 50000 curr count: 26426
2021/12/01 18:24:06 Waiting for dst bucket item count to get to: 50000 curr count: 26426
2021/12/01 18:24:11 Waiting for dst bucket item count to get to: 50000 curr count: 37456
2021/12/01 18:24:16 Waiting for dst bucket item count to get to: 50000 curr count: 37456
2021/12/01 18:24:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:24:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:24:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:24:21 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:26 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:31 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:36 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/01 18:24:46 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/01 18:24:46 No eventing-con process running
2021/12/01 18:24:46 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/01 18:24:48 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:24:53 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:24:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:25:00 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:25:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:25:10 Waiting for dst bucket item count to get to: 0 curr count: 10365
2021/12/01 18:25:15 Waiting for dst bucket item count to get to: 0 curr count: 10365
2021/12/01 18:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.80s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/01 18:25:25 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:25:25 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/01 18:25:25 No eventing-con process running
2021/12/01 18:25:25 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/01 18:25:27 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:25:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:25:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:25:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:25:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:25:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:25:44 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:25:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 18:25:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/01 18:25:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/01 18:26:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/01 18:26:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/01 18:26:05 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/01 18:26:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:26:05 Starting bucket ops to source bucket
2021/12/01 18:26:06 Removing node(s): [] from the cluster
2021/12/01 18:26:11 Rebalance progress: 12.5
2021/12/01 18:26:16 Rebalance progress: 12.5
2021/12/01 18:26:21 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:26:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:26:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:26:26 Rebalance progress: 12.5
2021/12/01 18:26:31 Rebalance progress: 12.5
2021/12/01 18:26:36 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:26:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:26:42 Rebalance progress: 12.5
2021/12/01 18:26:47 Rebalance progress: 12.5
2021/12/01 18:26:52 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:52 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:26:52 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:26:57 Waiting for dst bucket item count to get to: 50000 curr count: 36071
2021/12/01 18:27:02 Waiting for dst bucket item count to get to: 50000 curr count: 36071
2021/12/01 18:27:07 Waiting for dst bucket item count to get to: 50000 curr count: 42093
2021/12/01 18:27:12 Waiting for dst bucket item count to get to: 50000 curr count: 42093
2021/12/01 18:27:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:27:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:27:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:27:17 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:22 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:27 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:32 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:37 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:42 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/01 18:27:42 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/01 18:27:42 No eventing-con process running
2021/12/01 18:27:42 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/01 18:27:44 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:27:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:27:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:27:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:27:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:28:01 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:28:06 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:28:11 Waiting for dst bucket item count to get to: 0 curr count: 34603
2021/12/01 18:28:16 Waiting for dst bucket item count to get to: 0 curr count: 34603
2021/12/01 18:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.50s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/01 18:28:26 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:28:26 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/01 18:28:26 No eventing-con process running
2021/12/01 18:28:26 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/01 18:28:27 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:28:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:28:34 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:28:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:28:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:28:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:28:44 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:28:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 18:28:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/01 18:28:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/01 18:29:00 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/01 18:29:05 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/01 18:29:05 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/01 18:29:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:29:05 Starting bucket ops to source bucket
2021/12/01 18:29:06 Removing node(s): [] from the cluster
2021/12/01 18:29:11 Rebalance progress: 12.5
2021/12/01 18:29:16 Rebalance progress: 12.5
2021/12/01 18:29:21 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:29:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:29:21 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:29:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:29:26 Rebalance progress: 12.5
2021/12/01 18:29:31 Rebalance progress: 12.5
2021/12/01 18:29:36 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:29:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:29:36 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:29:36 Waiting for src bucket item count to get to: 50000 curr count: 33411
2021/12/01 18:29:41 Waiting for src bucket item count to get to: 50000 curr count: 47547
2021/12/01 18:29:46 Waiting for src bucket item count to get to: 50000 curr count: 47547
2021/12/01 18:29:56 Waiting for dst bucket item count to get to: 50000 curr count: 48235
2021/12/01 18:30:01 Waiting for dst bucket item count to get to: 50000 curr count: 48235
2021/12/01 18:30:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:30:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:30:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:30:06 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:11 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:16 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:21 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:26 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:31 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/01 18:30:31 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/01 18:30:31 No eventing-con process running
2021/12/01 18:30:31 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/01 18:30:33 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:30:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:30:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:30:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:30:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:30:50 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:30:55 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:30:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (159.14s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/01 18:31:00 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:31:00 Removing node(s): [] from the cluster
2021/12/01 18:31:05 Rebalance progress: 100
2021/12/01 18:31:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:31:10 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/01 18:31:10 No eventing-con process running
2021/12/01 18:31:10 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/01 18:31:12 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:31:17 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:31:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:31:19 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:31:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:31:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:31:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:31:29 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:31:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 18:31:35 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/01 18:31:40 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/01 18:31:45 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/01 18:31:50 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/01 18:31:50 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/01 18:31:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:31:50 Starting bucket ops to source bucket
2021/12/01 18:31:55 Rebalance progress: 12.5
2021/12/01 18:32:00 Rebalance progress: 12.5
2021/12/01 18:32:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:32:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:32:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:32:05 Waiting for src bucket item count to get to: 50000 curr count: 13790
2021/12/01 18:32:10 Waiting for src bucket item count to get to: 50000 curr count: 28881
2021/12/01 18:32:15 Waiting for src bucket item count to get to: 50000 curr count: 28881
2021/12/01 18:32:20 Waiting for src bucket item count to get to: 50000 curr count: 42977
2021/12/01 18:32:25 Waiting for src bucket item count to get to: 50000 curr count: 42977
2021/12/01 18:32:35 Waiting for dst bucket item count to get to: 50000 curr count: 31173
2021/12/01 18:32:40 Waiting for dst bucket item count to get to: 50000 curr count: 31173
2021/12/01 18:32:45 Waiting for dst bucket item count to get to: 50000 curr count: 46006
2021/12/01 18:32:50 Waiting for dst bucket item count to get to: 50000 curr count: 46006
2021/12/01 18:32:55 Waiting for dst bucket item count to get to: 50000 curr count: 46006
2021/12/01 18:32:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:32:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:32:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:33:00 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:05 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:10 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:15 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:20 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:25 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/01 18:33:25 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/01 18:33:25 No eventing-con process running
2021/12/01 18:33:26 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/01 18:33:27 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:33:32 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:33:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:33:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:33:39 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:33:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:33:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (174.44s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/01 18:33:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:33:59 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/01 18:33:59 No eventing-con process running
2021/12/01 18:33:59 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/01 18:34:01 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:34:06 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:34:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:34:08 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:34:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:34:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:34:18 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:34:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 18:34:24 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/01 18:34:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/01 18:34:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/01 18:34:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/01 18:34:39 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/01 18:34:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/01 18:34:39 Starting bucket ops to source bucket
2021/12/01 18:34:39 Removing node(s): [] from the cluster
2021/12/01 18:34:44 Rebalance progress: 12.5
2021/12/01 18:34:49 Rebalance progress: 12.5
2021/12/01 18:34:54 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:34:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:34:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:34:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 18:34:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:35:00 Rebalance progress: 12.5
2021/12/01 18:35:05 Rebalance progress: 12.5
2021/12/01 18:35:10 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:35:10 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:35:10 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:35:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:35:15 Rebalance progress: 12.5
2021/12/01 18:35:20 Rebalance progress: 12.5
2021/12/01 18:35:25 Rebalance progress: 100
2021/12/01 18:35:30 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:35:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:35:30 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:35:35 Waiting for dst bucket item count to get to: 50000 curr count: 29786
2021/12/01 18:35:40 Waiting for dst bucket item count to get to: 50000 curr count: 46629
2021/12/01 18:35:45 Waiting for dst bucket item count to get to: 50000 curr count: 46629
2021/12/01 18:35:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:35:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:35:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:35:50 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:35:55 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:36:00 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:36:06 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:36:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:36:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/01 18:36:16 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/01 18:36:16 No eventing-con process running
2021/12/01 18:36:16 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/01 18:36:17 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:36:22 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:36:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:36:29 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:36:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:36:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:36:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.03s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/01 18:36:49 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:36:49 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/01 18:36:49 No eventing-con process running
2021/12/01 18:36:49 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/01 18:36:51 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:36:56 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:36:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:36:58 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:37:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:37:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:37:08 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:37:08 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:37:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/01 18:37:14 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/01 18:37:19 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/01 18:37:24 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/01 18:37:29 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/01 18:37:29 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/01 18:37:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:37:29 Starting bucket ops to source bucket
2021/12/01 18:37:30 Removing node(s): [] from the cluster
2021/12/01 18:37:35 Rebalance progress: 12.5
2021/12/01 18:37:40 Rebalance progress: 12.5
2021/12/01 18:37:45 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:37:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:37:45 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:37:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:37:50 Rebalance progress: 12.5
2021/12/01 18:37:55 Rebalance progress: 12.5
2021/12/01 18:38:00 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:38:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:38:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:38:00 Waiting for src bucket item count to get to: 50000 curr count: 30239
2021/12/01 18:38:05 Waiting for src bucket item count to get to: 50000 curr count: 44147
2021/12/01 18:38:10 Waiting for src bucket item count to get to: 50000 curr count: 44147
2021/12/01 18:38:20 Waiting for dst bucket item count to get to: 50000 curr count: 31950
2021/12/01 18:38:25 Waiting for dst bucket item count to get to: 50000 curr count: 31950
2021/12/01 18:38:30 Waiting for dst bucket item count to get to: 50000 curr count: 48825
2021/12/01 18:38:35 Waiting for dst bucket item count to get to: 50000 curr count: 48825
2021/12/01 18:38:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:38:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:38:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:38:40 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:38:45 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:38:50 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:38:55 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:39:00 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:39:05 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/01 18:39:05 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/01 18:39:05 No eventing-con process running
2021/12/01 18:39:05 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/01 18:39:07 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:39:12 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:39:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:39:19 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:39:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:39:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:39:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.75s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/01 18:39:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:39:34 Removing node(s): [] from the cluster
2021/12/01 18:39:39 Rebalance progress: 100
2021/12/01 18:39:44 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:39:44 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/01 18:39:44 No eventing-con process running
2021/12/01 18:39:44 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/01 18:39:46 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:39:51 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:39:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:39:53 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:39:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:39:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:40:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:40:03 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:40:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/01 18:40:09 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/01 18:40:14 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/01 18:40:19 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/01 18:40:24 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/01 18:40:24 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/01 18:40:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:40:24 Starting bucket ops to source bucket
2021/12/01 18:40:29 Rebalance progress: 12.5
2021/12/01 18:40:34 Rebalance progress: 12.5
2021/12/01 18:40:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:40:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:40:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:40:39 Waiting for src bucket item count to get to: 50000 curr count: 7168
2021/12/01 18:40:44 Waiting for src bucket item count to get to: 50000 curr count: 22654
2021/12/01 18:40:49 Waiting for src bucket item count to get to: 50000 curr count: 22654
2021/12/01 18:40:54 Waiting for src bucket item count to get to: 50000 curr count: 37014
2021/12/01 18:40:59 Waiting for src bucket item count to get to: 50000 curr count: 37014
2021/12/01 18:41:09 Waiting for dst bucket item count to get to: 50000 curr count: 30106
2021/12/01 18:41:14 Waiting for dst bucket item count to get to: 50000 curr count: 30106
2021/12/01 18:41:19 Waiting for dst bucket item count to get to: 50000 curr count: 30106
2021/12/01 18:41:24 Waiting for dst bucket item count to get to: 50000 curr count: 46715
2021/12/01 18:41:29 Waiting for dst bucket item count to get to: 50000 curr count: 46715
2021/12/01 18:41:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:41:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:41:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:41:34 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:41:39 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:41:45 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:41:50 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:41:55 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/01 18:41:55 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/01 18:41:55 No eventing-con process running
2021/12/01 18:41:55 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/01 18:41:57 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:42:02 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:42:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:42:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:42:08 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:42:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:42:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:42:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.46s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/01 18:42:28 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:42:28 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/01 18:42:28 No eventing-con process running
2021/12/01 18:42:28 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/01 18:42:30 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:42:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:42:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:42:37 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:42:42 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:42:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:42:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:42:47 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:42:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/01 18:42:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/01 18:42:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/01 18:43:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/01 18:43:08 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/01 18:43:08 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/01 18:43:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:43:08 Starting bucket ops to source bucket
2021/12/01 18:43:09 Removing node(s): [] from the cluster
2021/12/01 18:43:14 Rebalance progress: 12.5
2021/12/01 18:43:19 Rebalance progress: 12.5
2021/12/01 18:43:24 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:24 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:43:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:43:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:43:29 Rebalance progress: 12.5
2021/12/01 18:43:34 Rebalance progress: 12.5
2021/12/01 18:43:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:43:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:43:44 Rebalance progress: 12.5
2021/12/01 18:43:49 Rebalance progress: 12.5
2021/12/01 18:43:54 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:43:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:43:59 Waiting for dst bucket item count to get to: 50000 curr count: 32204
2021/12/01 18:44:04 Waiting for dst bucket item count to get to: 50000 curr count: 32204
2021/12/01 18:44:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/01 18:44:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/01 18:44:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:44:09 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:15 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:20 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:30 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:35 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/01 18:44:35 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/01 18:44:35 No eventing-con process running
2021/12/01 18:44:35 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/01 18:44:37 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:44:42 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:44:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:44:48 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:44:53 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:44:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/01 18:44:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (160.00s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/01 18:45:08 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:45:08 App: bucket_op_on_update got un-deployed
2021/12/01 18:45:08 No eventing-con process running
2021/12/01 18:45:08 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:45:10 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:45:15 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:45:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:45:17 Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:45:22 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:45:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:45:27 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:45:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 18:45:33 Starting bucket ops to source bucket
2021/12/01 18:45:38 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:45:43 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:45:48 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:45:48 App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:45:48 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:45:48 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:45:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:45:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:45:54 Rebalance progress: 12.5
2021/12/01 18:45:59 Rebalance progress: 12.5
2021/12/01 18:46:04 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:46:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:46:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:46:09 Rebalance progress: 12.5
2021/12/01 18:46:14 Rebalance progress: 12.5
2021/12/01 18:46:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:46:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 18:46:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:46:25 Rebalance progress: 12.5
2021/12/01 18:46:30 Rebalance progress: 12.5
2021/12/01 18:46:35 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:35 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:46:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:46:40 Rebalance progress: 12.5
2021/12/01 18:46:45 Rebalance progress: 12.5
2021/12/01 18:46:50 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:46:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:46:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:46:55 Rebalance progress: 12.5
2021/12/01 18:47:00 Rebalance progress: 12.5
2021/12/01 18:47:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:47:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:47:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:47:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 18:47:10 Rebalance progress: 12.5
2021/12/01 18:47:15 Rebalance progress: 12.5
2021/12/01 18:47:20 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:47:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:47:20 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:47:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:47:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:47:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:47:50 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:47:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 18:47:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:47:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:48:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10679
2021/12/01 18:48:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10679
2021/12/01 18:48:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:48:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:48:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:48:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10679
2021/12/01 18:48:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.61s)
=== RUN   TestKVForceFailover
2021/12/01 18:48:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:48:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:48:34 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:48:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:48:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:48:41 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:48:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:48:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:48:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:48:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:48:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:48:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:48:53 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:48:53 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:48:53 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:48:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 18:48:54 Starting bucket ops to source bucket
2021/12/01 18:48:59 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:49:04 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:49:09 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:49:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:49:14 App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:49:14 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:49:14 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:49:14 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/01 18:49:34 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/01 18:49:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/01 18:49:40 Rebalance progress: 12.5
2021/12/01 18:49:45 Rebalance progress: 12.5
2021/12/01 18:49:50 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:49:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:49:50 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:49:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/01 18:49:55 Rebalance progress: 12.5
2021/12/01 18:50:00 Rebalance progress: 12.5
2021/12/01 18:50:05 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:50:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:50:05 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:50:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:50:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:50:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:50:35 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:50:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 18:50:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:50:42 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:50:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7041
2021/12/01 18:50:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:50:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:50:54 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:50:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7041
2021/12/01 18:50:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.55s)
=== RUN   TestKVHardFailoverRecovery
2021/12/01 18:51:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:51:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:51:09 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:51:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:51:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:51:16 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:51:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:51:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:51:22 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:51:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:51:27 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:51:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 18:51:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 18:51:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:51:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:51:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:51:48 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:51:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:51:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:51:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:51:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:51:54 Rebalance progress: 12.5
2021/12/01 18:51:59 Rebalance progress: 12.5
2021/12/01 18:52:04 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:52:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:52:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:52:04 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/01 18:53:04 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/01 18:53:09 Rebalance progress: 12.5
2021/12/01 18:53:14 Rebalance progress: 12.5
2021/12/01 18:53:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:53:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:53:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:53:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:53:24 Rebalance progress: 12.5
2021/12/01 18:53:29 Rebalance progress: 12.5
2021/12/01 18:53:34 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:53:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:53:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:53:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:53:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:53:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:53:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:53:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:53:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:54:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:54:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:54:04 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:54:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 18:54:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:54:11 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:54:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12048
2021/12/01 18:54:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:54:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:54:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:54:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12048
2021/12/01 18:54:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.39s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/01 18:54:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:54:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:54:38 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:54:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:54:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:54:45 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:54:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:54:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:54:52 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:54:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:54:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:54:57 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:54:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 18:55:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 18:55:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:55:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:55:18 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:55:18 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:18 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:18 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:55:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:55:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:55:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 18:55:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:55:24 Rebalance progress: 12.5
2021/12/01 18:55:29 Rebalance progress: 12.5
2021/12/01 18:55:34 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:34 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:55:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/01 18:55:39 Rebalance progress: 12.5
2021/12/01 18:55:44 Rebalance progress: 12.5
2021/12/01 18:55:49 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:55:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:55:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:55:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:55:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:56:19 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:56:20 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 18:56:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:56:26 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:56:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4614
2021/12/01 18:56:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4614
2021/12/01 18:56:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:56:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:56:43 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:56:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4614
2021/12/01 18:56:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.19s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/01 18:56:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:56:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:56:58 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:56:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:57:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:05 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:57:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:57:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:12 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:57:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:12 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:57:12 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 18:57:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 18:57:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 18:57:23 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:57:28 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:57:33 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 18:57:33 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:33 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:33 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:57:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 18:57:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 18:57:39 Rebalance progress: 12.5
2021/12/01 18:57:44 Rebalance progress: 12.5
2021/12/01 18:57:49 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:57:49 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:57:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 18:57:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 18:57:54 Rebalance progress: 12.5
2021/12/01 18:57:59 Rebalance progress: 12.5
2021/12/01 18:58:04 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:58:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:58:04 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:58:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 18:58:09 Rebalance progress: 12.5
2021/12/01 18:58:14 Rebalance progress: 12.5
2021/12/01 18:58:19 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:58:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:58:19 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 18:58:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 18:58:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:58:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:58:49 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:58:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 18:58:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:58:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:59:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6085
2021/12/01 18:59:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6085
2021/12/01 18:59:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:59:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:59:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:59:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6085
2021/12/01 18:59:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:59:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:59:23 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:59:23 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:59:23 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:59:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:59:30 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:59:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:59:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:59:37 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:59:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.39s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/01 18:59:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 18:59:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 18:59:47 eventingSetup.go:46: No eventing-con process running
2021/12/01 18:59:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 18:59:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 18:59:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 18:59:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 18:59:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:00 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:00:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:00:05 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:00:05 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/01 19:00:05 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:00:05 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/01 19:00:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:12 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:00:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:00:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:00:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:19 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:00:19 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:00:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:20 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:00:20 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:00:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/01 19:00:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 19:00:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:00:36 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:00:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:00:41 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/01 19:00:46 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:00:51 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:00:56 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:00:56 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:00:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:00:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 19:00:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:01:02 Rebalance progress: 12.5
2021/12/01 19:01:07 Rebalance progress: 12.5
2021/12/01 19:01:12 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:12 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:12 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:01:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 19:01:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:01:17 Rebalance progress: 12.5
2021/12/01 19:01:22 Rebalance progress: 12.5
2021/12/01 19:01:27 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:27 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:01:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/01 19:01:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:01:33 Rebalance progress: 12.5
2021/12/01 19:01:38 Rebalance progress: 12.5
2021/12/01 19:01:43 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:43 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:43 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:01:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 19:01:48 Rebalance progress: 12.5
2021/12/01 19:01:53 Rebalance progress: 12.5
2021/12/01 19:01:58 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:01:58 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:01:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 19:02:03 Rebalance progress: 12.5
2021/12/01 19:02:08 Rebalance progress: 12.5
2021/12/01 19:02:13 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:02:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:02:13 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:02:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/01 19:02:18 Rebalance progress: 12.5
2021/12/01 19:02:23 Rebalance progress: 12.5
2021/12/01 19:02:28 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:02:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:02:28 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:02:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:02:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:02:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:02:58 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:02:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 19:03:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:03:06 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:03:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7927
2021/12/01 19:03:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7927
2021/12/01 19:03:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:03:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:03:23 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:03:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12176
2021/12/01 19:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:03:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:03:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:38 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:44 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:49 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:54 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:59 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:03:59 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/01 19:03:59 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:03:59 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/01 19:04:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:06 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:04:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:12 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (275.78s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/01 19:04:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:04:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:04:22 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:04:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 19:04:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:29 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:04:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:36 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:04:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:04:41 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/01 19:04:41 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:04:41 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/01 19:04:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:04:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:55 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:55 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:55 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:04:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:04:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:04:56 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:04:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/01 19:05:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 19:05:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:05:12 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:05:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:05:17 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/01 19:05:22 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:05:27 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:05:32 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/01 19:05:32 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:05:32 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:05:32 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:05:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 19:05:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:05:37 Rebalance progress: 12.5
2021/12/01 19:05:42 Rebalance progress: 12.5
2021/12/01 19:05:47 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:05:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:05:47 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:05:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 19:05:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 19:05:53 Rebalance progress: 12.5
2021/12/01 19:05:58 Rebalance progress: 12.5
2021/12/01 19:06:03 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:06:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:06:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:06:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 19:06:08 Rebalance progress: 12.5
2021/12/01 19:06:13 Rebalance progress: 12.5
2021/12/01 19:06:18 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:06:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:06:18 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:06:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:06:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:06:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:06:48 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:06:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 19:06:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:06:57 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:07:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4827
2021/12/01 19:07:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4827
2021/12/01 19:07:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:07:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:07:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:07:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7604
2021/12/01 19:07:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:07:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:07:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:29 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:34 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:39 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:44 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:49 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/01 19:07:49 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/01 19:07:49 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:07:49 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/01 19:07:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:07:56 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:07:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:07:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:03 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:08:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.40s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/01 19:08:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:08:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:08:13 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:08:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 19:08:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:20 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:08:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:08:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:26 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:08:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:31 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:08:31 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:08:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 19:08:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 19:08:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:08:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:08:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:08:53 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:08:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:08:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 19:08:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:09:13 clusterSetup.go:165: Stopping rebalance
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:14 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:24 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:24 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:09:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:09:29 Rebalance progress: 12.5
2021/12/01 19:09:34 Rebalance progress: 12.5
2021/12/01 19:09:39 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:39 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:09:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 19:09:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 19:09:44 Rebalance progress: 12.5
2021/12/01 19:09:49 Rebalance progress: 12.5
2021/12/01 19:09:54 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:09:54 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:09:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 19:09:59 Rebalance progress: 12.5
2021/12/01 19:10:04 Rebalance progress: 12.5
2021/12/01 19:10:09 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:10:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:10:09 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:10:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:10:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:10:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:10:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:10:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:10:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:10:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:10:35 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:10:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 19:10:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:10:42 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:10:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9169
2021/12/01 19:10:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:10:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:10:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:10:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9169
2021/12/01 19:10:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.57s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/01 19:11:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:11:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:11:08 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:11:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/01 19:11:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:11:15 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:11:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:11:22 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:11:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:11:27 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:11:27 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:11:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 19:11:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/01 19:11:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:11:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:11:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:11:48 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:11:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:11:48 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:11:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 19:11:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:11:53 Rebalance progress: 12.5
2021/12/01 19:11:58 Rebalance progress: 12.5
2021/12/01 19:12:03 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:03 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:12:03 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:13 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:12:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:12:19 Rebalance progress: 12.5
2021/12/01 19:12:24 Rebalance progress: 12.5
2021/12/01 19:12:29 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:29 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:12:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/01 19:12:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/01 19:12:34 Rebalance progress: 12.5
2021/12/01 19:12:39 Rebalance progress: 12.5
2021/12/01 19:12:44 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:44 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:12:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/01 19:12:49 Rebalance progress: 12.5
2021/12/01 19:12:54 Rebalance progress: 12.5
2021/12/01 19:12:59 Rebalance progress: 100
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:12:59 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:12:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:13:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:13:30 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:13:30 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:13:30 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 19:13:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:13:36 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:13:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/01 19:13:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/01 19:13:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:13:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:13:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8604
2021/12/01 19:13:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.81s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/01 19:14:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/01 19:14:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/01 19:14:08 Rebalance progress: 100
2021/12/01 19:14:08 Failing over: [127.0.0.1:19001]
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:14:38 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:14:38 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/01 19:14:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/01 19:14:45 Removing node(s): [] from the cluster
2021/12/01 19:14:45 Starting bucket ops to source bucket
2021/12/01 19:14:50 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/01 19:14:55 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:15:00 Waiting for app: bucket_op_on_update to get deployed
2021/12/01 19:15:00 App: bucket_op_on_update got deployed
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:15:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:15:00 waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/01 19:15:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/01 19:15:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:15:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:15:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:15:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:15:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/01 19:15:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/01 19:15:25 eventingSetup.go:46: No eventing-con process running
2021/12/01 19:15:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/01 19:15:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:15:32 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:15:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1499
2021/12/01 19:15:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/01 19:15:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/01 19:15:44 clusterSetup.go:81: waitForIndexes giving up: err Get http://127.0.0.1:9108/getIndexStatus: dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/01 19:15:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1499
2021/12/01 19:15:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.41s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	11942.660s

Rebalance tests took 3 hours, 19 minutes and 6 seconds.

Testing Succeeded


Version: versions-01.12.2021-11.38.cfg
Build Log: make-01.12.2021-11.38.log
Server Log: logs-01.12.2021-11.38.tar.gz

Finished