crepererum · July 21, 2023 09:25
diff --git a/ingester_flight_api.log b/ingester_flight_api.log
 failures:

 ---- end_to_end_cases::ingester::ingester_flight_api stdout ----
 2023-07-21T09:13:19.016261Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:19.016290Z  INFO test_helpers_end_to_end::server_fixture: Server Ingester Logging to "/tmp/.tmpsmZ1LX"
 2023-07-21T09:13:19.016297Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:19.593062Z  INFO test_helpers_end_to_end::database: Initializing database... dsn=postgres://postgres@localhost/iox_shared schema_name=XDPhlKuZlNvCkAksheDD
 2023-07-21T09:13:19.599417Z DEBUG sqlx::query: summary="select exists(SELECT 1 from …" db.statement="\n\nselect\n  exists(\n    SELECT\n      1\n    from\n      pg_database\n    WHERE\n      datname = $1\n  )\n" rows_affected=0 rows_returned=1 elapsed=977.878µs
 2023-07-21T09:13:19.988388Z  INFO test_helpers_end_to_end: Running command: `INFLUXDB_IOX_BIND_ADDR=127.0.0.1:8191 INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost/iox_shared INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME=XDPhlKuZlNvCkAksheDD INFLUXDB_IOX_DB_DIR=/tmp/.tmpPMx6pk INFLUXDB_IOX_GRPC_BIND_ADDR=127.0.0.1:8190 INFLUXDB_IOX_OBJECT_STORE=file INFLUXDB_IOX_WAL_DIRECTORY=/tmp/.tmpsjT3o2 INFLUXDB_IOX_WAL_ROTATION_PERIOD_SECONDS=1 LOG_FILTER=debug,,hyper::proto::h1=info,h2=info INFLUXDB_IOX_BIND_ADDR="127.0.0.1:8191" INFLUXDB_IOX_CATALOG_DSN="postgres://postgres@localhost/iox_shared" INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME="XDPhlKuZlNvCkAksheDD" INFLUXDB_IOX_DB_DIR="/tmp/.tmpPMx6pk" INFLUXDB_IOX_GRPC_BIND_ADDR="127.0.0.1:8190" INFLUXDB_IOX_OBJECT_STORE="file" INFLUXDB_IOX_WAL_DIRECTORY="/tmp/.tmpsjT3o2" INFLUXDB_IOX_WAL_ROTATION_PERIOD_SECONDS="1" LOG_FILTER="debug,,hyper::proto::h1=info,h2=info" "/home/rust/project/target/debug/influxdb_iox" "run" "ingester"`
 2023-07-21T09:13:19.988780Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:19.988898Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:19.989082Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:19.990193Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:19.990266Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:19.990393Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:20.089941Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:20.090087Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:20.090296Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:20.190637Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:20.190780Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:20.190942Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8190
 2023-07-21T09:13:20.191679Z DEBUG reqwest::connect: starting new connection: http://127.0.0.1:8191/    
 2023-07-21T09:13:20.191728Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8191
 2023-07-21T09:13:20.191944Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:20.192286Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8191
 2023-07-21T09:13:20.194514Z  INFO test_helpers_end_to_end::server_fixture: Flight service Ingester is running
 2023-07-21T09:13:20.194714Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:8191)
 2023-07-21T09:13:20.194758Z  INFO test_helpers_end_to_end::server_fixture: Successfully got a response from Ingester HTTP: Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8191), path: "/health", query: None, fragment: None }, status: 200, headers: {"content-length": "2", "date": "Fri, 21 Jul 2023 09:13:20 GMT"} }
 2023-07-21T09:13:20.194864Z  INFO test_helpers_end_to_end::server_fixture: Successfully started TestServer Ingester (router_http: 127.0.0.1:8191 ingester_grpc: 127.0.0.1:8190 )
 2023-07-21T09:13:20.195003Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:20.195010Z  INFO test_helpers_end_to_end::server_fixture: Server Router Logging to "/tmp/.tmptiNkeW"
 2023-07-21T09:13:20.195015Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:20.830038Z  INFO test_helpers_end_to_end: Running command: `INFLUXDB_IOX_BIND_ADDR=127.0.0.1:8211 INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost/iox_shared INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME=XDPhlKuZlNvCkAksheDD INFLUXDB_IOX_DB_DIR=/tmp/.tmpPMx6pk INFLUXDB_IOX_GRPC_BIND_ADDR=127.0.0.1:8212 INFLUXDB_IOX_INGESTER_ADDRESSES=http://127.0.0.1:8190 INFLUXDB_IOX_OBJECT_STORE=file LOG_FILTER=debug,,hyper::proto::h1=info,h2=info INFLUXDB_IOX_BIND_ADDR="127.0.0.1:8211" INFLUXDB_IOX_CATALOG_DSN="postgres://postgres@localhost/iox_shared" INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME="XDPhlKuZlNvCkAksheDD" INFLUXDB_IOX_DB_DIR="/tmp/.tmpPMx6pk" INFLUXDB_IOX_GRPC_BIND_ADDR="127.0.0.1:8212" INFLUXDB_IOX_INGESTER_ADDRESSES="http://127.0.0.1:8190" INFLUXDB_IOX_OBJECT_STORE="file" LOG_FILTER="debug,,hyper::proto::h1=info,h2=info" "/home/rust/project/target/debug/influxdb_iox" "run" "router"`
 2023-07-21T09:13:20.844475Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8212
 2023-07-21T09:13:20.844645Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8212
 2023-07-21T09:13:20.844982Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to router at http://127.0.0.1:8212: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:20.845024Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8212
 2023-07-21T09:13:20.845119Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8212
 2023-07-21T09:13:20.845317Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to router at http://127.0.0.1:8212: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:20.957582Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8212
 2023-07-21T09:13:20.957759Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8212
 2023-07-21T09:13:20.963384Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8212
 2023-07-21T09:13:20.964702Z DEBUG reqwest::connect: starting new connection: http://127.0.0.1:8211/    
 2023-07-21T09:13:20.964775Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8211
 2023-07-21T09:13:20.965092Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:20.965631Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8211
 2023-07-21T09:13:20.971249Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:8211)
 2023-07-21T09:13:20.971315Z  INFO test_helpers_end_to_end::server_fixture: Successfully got a response from Router HTTP: Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8211), path: "/health", query: None, fragment: None }, status: 200, headers: {"content-length": "2", "date": "Fri, 21 Jul 2023 09:13:20 GMT"} }
 2023-07-21T09:13:20.973588Z  INFO test_helpers_end_to_end::server_fixture: CatalogService service Router is running
 2023-07-21T09:13:20.973623Z  INFO test_helpers_end_to_end::server_fixture: Successfully started TestServer Router (router_http: 127.0.0.1:8211 router_grpc: 127.0.0.1:8212 ingester_grpc: 127.0.0.1:8213 )
 2023-07-21T09:13:20.973756Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:20.973763Z  INFO test_helpers_end_to_end::server_fixture: Server Querier Logging to "/tmp/.tmphddWWb"
 2023-07-21T09:13:20.973769Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:22.369878Z  INFO test_helpers_end_to_end: Running command: `INFLUXDB_IOX_BIND_ADDR=127.0.0.1:8238 INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost/iox_shared INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME=XDPhlKuZlNvCkAksheDD INFLUXDB_IOX_DATAFUSION_CONFIG=iox.influxql_metadata_cutoff:1990-01-01T00:00:00Z INFLUXDB_IOX_DB_DIR=/tmp/.tmpPMx6pk INFLUXDB_IOX_GRPC_BIND_ADDR=127.0.0.1:8239 INFLUXDB_IOX_INGESTER_ADDRESSES=http://127.0.0.1:8190 INFLUXDB_IOX_NUM_QUERY_THREADS=4 INFLUXDB_IOX_OBJECT_STORE=file LOG_FILTER=debug,,hyper::proto::h1=info,h2=info INFLUXDB_IOX_BIND_ADDR="127.0.0.1:8238" INFLUXDB_IOX_CATALOG_DSN="postgres://postgres@localhost/iox_shared" INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME="XDPhlKuZlNvCkAksheDD" INFLUXDB_IOX_DATAFUSION_CONFIG="iox.influxql_metadata_cutoff:1990-01-01T00:00:00Z" INFLUXDB_IOX_DB_DIR="/tmp/.tmpPMx6pk" INFLUXDB_IOX_GRPC_BIND_ADDR="127.0.0.1:8239" INFLUXDB_IOX_INGESTER_ADDRESSES="http://127.0.0.1:8190" INFLUXDB_IOX_NUM_QUERY_THREADS="4" INFLUXDB_IOX_OBJECT_STORE="file" LOG_FILTER="debug,,hyper::proto::h1=info,h2=info" "/home/rust/project/target/debug/influxdb_iox" "run" "querier"`
 2023-07-21T09:13:22.370413Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8239
 2023-07-21T09:13:22.370498Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8239
 2023-07-21T09:13:22.370680Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to querier at http://127.0.0.1:8239: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:22.371808Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8239
 2023-07-21T09:13:22.371886Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8239
 2023-07-21T09:13:22.372047Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to querier at http://127.0.0.1:8239: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:22.471388Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8239
 2023-07-21T09:13:22.471577Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8239
 2023-07-21T09:13:22.471808Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8239
 2023-07-21T09:13:22.472680Z DEBUG reqwest::connect: starting new connection: http://127.0.0.1:8238/    
 2023-07-21T09:13:22.472739Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8238
 2023-07-21T09:13:22.472946Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.473303Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8238
 2023-07-21T09:13:22.474866Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:8238)
 2023-07-21T09:13:22.474933Z  INFO test_helpers_end_to_end::server_fixture: Successfully got a response from Querier HTTP: Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8238), path: "/health", query: None, fragment: None }, status: 200, headers: {"content-length": "2", "date": "Fri, 21 Jul 2023 09:13:22 GMT"} }
 2023-07-21T09:13:22.518468Z  INFO test_helpers_end_to_end::server_fixture: Flight service Querier is running
 2023-07-21T09:13:22.518518Z  INFO test_helpers_end_to_end::server_fixture: Successfully started TestServer Querier (router_http: 127.0.0.1:8238 querier_grpc: 127.0.0.1:8239 )
 2023-07-21T09:13:22.518706Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8211
 2023-07-21T09:13:22.518982Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8211
 2023-07-21T09:13:22.578698Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:8211)
 2023-07-21T09:13:22.579062Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.584235Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.587916Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.599403Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.603570Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.654675Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.776429Z  INFO test_helpers_end_to_end::server_fixture: Killed child politely
 2023-07-21T09:13:22.776489Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:22.776497Z  INFO test_helpers_end_to_end::server_fixture: Server Ingester Logging to "/tmp/.tmpsmZ1LX"
 2023-07-21T09:13:22.776507Z  INFO test_helpers_end_to_end::server_fixture: ****************
 2023-07-21T09:13:22.776923Z DEBUG hyper::client::service: connection error: hyper::Error(Io, Custom { kind: BrokenPipe, error: "connection closed because of a broken pipe" })
 2023-07-21T09:13:23.113531Z  INFO test_helpers_end_to_end: Running command: `INFLUXDB_IOX_BIND_ADDR=127.0.0.1:8191 INFLUXDB_IOX_CATALOG_DSN=postgres://postgres@localhost/iox_shared INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME=XDPhlKuZlNvCkAksheDD INFLUXDB_IOX_DB_DIR=/tmp/.tmpPMx6pk INFLUXDB_IOX_GRPC_BIND_ADDR=127.0.0.1:8190 INFLUXDB_IOX_OBJECT_STORE=file INFLUXDB_IOX_WAL_DIRECTORY=/tmp/.tmpsjT3o2 INFLUXDB_IOX_WAL_ROTATION_PERIOD_SECONDS=1 LOG_FILTER=debug,,hyper::proto::h1=info,h2=info INFLUXDB_IOX_BIND_ADDR="127.0.0.1:8191" INFLUXDB_IOX_CATALOG_DSN="postgres://postgres@localhost/iox_shared" INFLUXDB_IOX_CATALOG_POSTGRES_SCHEMA_NAME="XDPhlKuZlNvCkAksheDD" INFLUXDB_IOX_DB_DIR="/tmp/.tmpPMx6pk" INFLUXDB_IOX_GRPC_BIND_ADDR="127.0.0.1:8190" INFLUXDB_IOX_OBJECT_STORE="file" INFLUXDB_IOX_WAL_DIRECTORY="/tmp/.tmpsjT3o2" INFLUXDB_IOX_WAL_ROTATION_PERIOD_SECONDS="1" LOG_FILTER="debug,,hyper::proto::h1=info,h2=info" "/home/rust/project/target/debug/influxdb_iox" "run" "ingester"`
 2023-07-21T09:13:23.152750Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.152922Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.153232Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.153273Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.153363Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.153623Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.257585Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.257773Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.258100Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.353599Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.353791Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.354097Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.453500Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.453672Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.453970Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.553317Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.553517Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.553813Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.653352Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.653586Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.653862Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.756276Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.756540Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.756972Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.857558Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.857693Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.857958Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:23.953256Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:23.953367Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:23.954398Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.055186Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.055328Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.055585Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.155167Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.155300Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.155559Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.253903Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.254049Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.254299Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.355264Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.355399Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.355677Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.453016Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.453159Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.453457Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.553860Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.554038Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.554300Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.653599Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.653779Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.654048Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.753585Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.753776Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.754283Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.859220Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.859408Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.859713Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:24.953054Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:24.953170Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:24.953451Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.053815Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.053994Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.054426Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.157149Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.157314Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.157610Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.254320Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.254479Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.254751Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.353171Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.353354Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.353685Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.453415Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.453638Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.453920Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.553324Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.553523Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.553808Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.653103Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.653243Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.653555Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.755683Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.755870Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.756175Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.853733Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.853867Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.854124Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:25.953563Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:25.953744Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:25.954021Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.053435Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.053612Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.053868Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.154660Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.154812Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.155064Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.253495Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.253669Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.253936Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.353286Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.353483Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.353737Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.461413Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.462020Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.462317Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.553633Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.553748Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.553957Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.653314Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.653500Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.653766Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.757373Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.757580Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.757921Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.853354Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.853624Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.853899Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:26.957569Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:26.957746Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:26.958070Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.057595Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.057765Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.058018Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.153398Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.153612Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.153922Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.261571Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.261744Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.262031Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.353453Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.353639Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.353931Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.453338Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.453561Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.453857Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.553242Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.553400Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.553686Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.653785Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.653955Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.654238Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.754814Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.758905Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.759099Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.853587Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.853769Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.854061Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:27.961540Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:27.961708Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:27.962042Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.053402Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.053582Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.053832Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.153577Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.153764Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.154051Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.253639Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.253820Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.254107Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.353475Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.353654Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.353928Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.453316Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.453529Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.453848Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.553746Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.553923Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.554196Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.653555Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.653731Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.653997Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.768874Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.769058Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.769346Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.853694Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.853882Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.854198Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:28.953613Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:28.953791Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:28.954055Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.053384Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.053561Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.053828Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.153703Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.153894Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.154211Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.253539Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.253699Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.253963Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.353730Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.353927Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.354259Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.456745Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.456922Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.457200Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.553584Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.553764Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.554067Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.653498Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.653681Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.653983Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.753364Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.753569Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.753838Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.853681Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.853872Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.854190Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:29.953624Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:29.953802Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:29.954085Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.053514Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.053696Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.053982Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.153404Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.153628Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.154838Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.253477Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.253654Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.253934Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.353354Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.353562Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.353839Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.453214Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.453399Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.453735Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.553889Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.554028Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.554268Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.653590Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.653768Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.654049Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.753472Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.753657Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.753948Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.853332Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.853571Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.853877Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:30.953268Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:30.953498Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:30.953821Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.053256Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.053486Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.053794Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.153182Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.153360Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.153928Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.253348Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.253604Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.253878Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.353676Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.353853Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.354127Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.453546Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.453720Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.453982Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.557588Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.557775Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.558074Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.653577Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.653758Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.654061Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.753514Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.753694Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.753990Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.853711Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.853893Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.854176Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:31.957586Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:31.957769Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:31.958048Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.053497Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.053679Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.053953Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.153491Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.154334Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.158614Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.253059Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.253245Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.254693Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.353723Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.353872Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.354114Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.453486Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.453670Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.453988Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.553375Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.553574Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.553848Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.661579Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.661765Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.662047Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.757543Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.757712Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.757981Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.853200Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.853376Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.853699Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:32.953049Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:32.953300Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:32.953654Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:33.053019Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:33.053107Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:33.053293Z  INFO test_helpers_end_to_end::server_fixture: wait_until_ready: can not yet connect: Cannot connect to ingester at http://127.0.0.1:8190: Connection error: transport error (error trying to connect: tcp connect error: Connection refused (os error 111))
 2023-07-21T09:13:33.155307Z  INFO test_helpers_end_to_end::server_fixture: Creating gRPC channel to http://127.0.0.1:8190
 2023-07-21T09:13:33.155447Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 thread 'end_to_end_cases::ingester::ingester_flight_api' panicked at 'timeout waiting for task to join: Elapsed(())', /home/rust/project/test_helpers/src/timeout.rs:23:14
 stack backtrace:
   0: rust_begin_unwind
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/result.rs:1651:5
   3: core::result::Result<T,E>::expect
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/result.rs:1033:23
   4: test_helpers::timeout::FutureTimeout::with_timeout_panic::{{closure}}
             at /home/rust/project/test_helpers/src/timeout.rs:21:9
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/future/future.rs:125:9
   6: test_helpers_end_to_end::server_fixture::TestServer::wait_until_ready::{{closure}}
             at /home/rust/project/test_helpers_end_to_end/src/server_fixture.rs:406:10
   7: test_helpers_end_to_end::server_fixture::ServerFixture::restart_server::{{closure}}
             at /home/rust/project/test_helpers_end_to_end/src/server_fixture.rs:72:53
   8: test_helpers_end_to_end::mini_cluster::MiniCluster::restart_ingesters::{{closure}}
             at /home/rust/project/test_helpers_end_to_end/src/mini_cluster.rs:332:54
   9: end_to_end::end_to_end_cases::ingester::ingester_flight_api::{{closure}}
             at ./tests/end_to_end_cases/ingester.rs:190:33
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/future/future.rs:125:9
  11: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/future/future.rs:125:9
  13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:651:57
  14: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107:5
  15: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73:5
  16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:651:25
  17: tokio::runtime::scheduler::current_thread::Context::enter
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:410:19
  18: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:650:36
  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:729:68
  20: tokio::runtime::context::scoped::Scoped<T>::set
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/scoped.rs:40:9
  21: tokio::runtime::context::set_scheduler::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:26
  22: std::thread::local::LocalKey<T>::try_with
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/thread/local.rs:270:16
  23: std::thread::local::LocalKey<T>::with
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/thread/local.rs:246:9
  24: tokio::runtime::context::set_scheduler
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:9
  25: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:729:27
  26: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:638:19
  27: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:175:28
  28: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65:16
  29: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:167:9
  30: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/runtime.rs:311:47
  31: end_to_end::end_to_end_cases::ingester::ingester_flight_api
             at ./tests/end_to_end_cases/ingester.rs:208:5
  32: end_to_end::end_to_end_cases::ingester::ingester_flight_api::{{closure}}
             at ./tests/end_to_end_cases/ingester.rs:113:32
  33: core::ops::function::FnOnce::call_once
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:250:5
  34: core::ops::function::FnOnce::call_once
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:250:5
 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
 2023-07-21T09:13:33.156136Z  WARN test_helpers_end_to_end::server_fixture: Server process exited: exit status: 1
 2023-07-21T09:13:33.156159Z  INFO test_helpers_end_to_end::server_fixture: Error sending SIGTERM to child: ESRCH: No such process
 2023-07-21T09:13:33.156173Z  INFO test_helpers_end_to_end::server_fixture: Cannot terminate child politely, using SIGKILL...
 2023-07-21T09:13:33.156179Z  INFO test_helpers_end_to_end::server_fixture: Error sending SIGKILL to child: ESRCH: No such process
 2023-07-21T09:13:33.156188Z  INFO test_helpers_end_to_end::server_fixture: Cannot wait for child: ECHILD: No child processes
 2023-07-21T09:13:33.156214Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.156220Z  INFO test_helpers_end_to_end: Start Ingester Output
 2023-07-21T09:13:33.156226Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:20.020520Z  INFO clap_blocks::catalog_dsn: Catalog: Postgres postgres_schema_name=XDPhlKuZlNvCkAksheDD
 2023-07-21T09:13:20.034380Z  INFO iox_catalog::postgres: connected to config store application_name=ingester
 2023-07-21T09:13:20.035190Z DEBUG datafusion_execution::memory_pool::pool: Created new GreedyMemoryPool(pool_size=18446744073709551615)    
 2023-07-21T09:13:20.035287Z  INFO clap_blocks::object_store: Object Store db_dir="/tmp/.tmpPMx6pk" object_store_type="Directory"
 2023-07-21T09:13:20.056369Z  INFO wal: Initalizing Write Ahead Log (WAL) wal_dir="/tmp/.tmpsjT3o2"
 2023-07-21T09:13:20.065241Z  INFO ingester::persist::handle: initialised persist task n_workers=5 persist_queue_depth=250
 2023-07-21T09:13:20.124620Z  INFO ingester::init::wal_replay: no wal replay files found
 2023-07-21T09:13:20.125053Z  INFO influxdb_iox::commands::run::ingester: starting ingester
 2023-07-21T09:13:20.129495Z  INFO influxdb_iox::commands::run::main: InfluxDB IOx server starting git_hash=e67d138dd609642a8070c831bddd6b80ca6563b5 version=0.1.0 uuid=76bbd9b8-d634-4edf-8ba4-9249f97d4442 num_cpus=16 build_malloc_conf=�
 2023-07-21T09:13:20.129979Z DEBUG influxdb_iox::commands::run::main: wait for handle server_type=Ingester
 2023-07-21T09:13:20.130035Z  INFO influxdb_iox::commands::run::main: Binding gRPC services grpc_bind_address=SocketAddr(127.0.0.1:8190) server_type=Ingester
 2023-07-21T09:13:20.130296Z  INFO ioxd_common: bound gRPC listener local_addr=127.0.0.1:8190
 2023-07-21T09:13:20.130329Z  INFO influxdb_iox::commands::run::main: Completed bind of gRPC, binding http http_bind_address=SocketAddr(127.0.0.1:8191) server_type=Ingester
 2023-07-21T09:13:20.130382Z  INFO ioxd_common: bound HTTP listener bind_addr=127.0.0.1:8191
 2023-07-21T09:13:20.130429Z  INFO ioxd_common: gRPC server listening server_type=Ingester
 2023-07-21T09:13:20.130452Z  INFO ioxd_common: InfluxDB IOx Ingester server ready    
 2023-07-21T09:13:20.156907Z  INFO ioxd_common: HTTP server listening server_type=Ingester
 2023-07-21T09:13:20.194133Z DEBUG ioxd_common::http: Processing request request=Request { method: GET, uri: /health, version: HTTP/1.1, headers: {"accept": "*/*", "host": "127.0.0.1:8191"}, body: Body(Empty) }
 2023-07-21T09:13:20.194219Z DEBUG ioxd_common::http: Successfully processed request response=Response { status: 200, version: HTTP/1.1, headers: {}, body: Body(Full(b"OK")) }
 2023-07-21T09:13:21.126196Z  INFO ingester::wal::rotate_task: rotating wal file
 2023-07-21T09:13:21.127071Z DEBUG ingester::wal::rotate_task: rotated wal closed_id=0 segment_bytes=16 n_ops=0
 2023-07-21T09:13:21.127311Z DEBUG ingester::persist::drain_buffer: queued all non-empty partitions for persist n_partitions=0
 2023-07-21T09:13:21.127344Z DEBUG ingester::wal::rotate_task: partitions persisted closed_id=0
 2023-07-21T09:13:21.127374Z DEBUG ingester::wal::reference_tracker::actor: notified of new segment file segment_id=0 sequence_number_set=SequenceNumberSet(Treemap<[]>)
 2023-07-21T09:13:21.127504Z DEBUG ingester::wal::reference_tracker::actor: immediately dropping empty segment file n=0
 2023-07-21T09:13:21.127524Z  INFO ingester::wal::reference_tracker::actor: deleted fully-persisted wal segment id=0
 2023-07-21T09:13:21.127609Z DEBUG ingester::wal::reference_tracker::actor: updated reference state num_files=0 files=[] persisted_set_len=0
 2023-07-21T09:13:22.141792Z  INFO ingester::wal::rotate_task: rotating wal file
 2023-07-21T09:13:22.142337Z DEBUG ingester::wal::rotate_task: rotated wal closed_id=1 segment_bytes=16 n_ops=0
 2023-07-21T09:13:22.142566Z DEBUG ingester::persist::drain_buffer: queued all non-empty partitions for persist n_partitions=0
 2023-07-21T09:13:22.142600Z DEBUG ingester::wal::rotate_task: partitions persisted closed_id=1
 2023-07-21T09:13:22.142649Z DEBUG ingester::wal::reference_tracker::actor: notified of new segment file segment_id=1 sequence_number_set=SequenceNumberSet(Treemap<[]>)
 2023-07-21T09:13:22.142731Z DEBUG ingester::wal::reference_tracker::actor: immediately dropping empty segment file n=0
 2023-07-21T09:13:22.142747Z  INFO ingester::wal::reference_tracker::actor: deleted fully-persisted wal segment id=1
 2023-07-21T09:13:22.142838Z DEBUG ingester::wal::reference_tracker::actor: updated reference state num_files=0 files=[] persisted_set_len=0
 2023-07-21T09:13:22.571647Z DEBUG ingester::buffer_tree::partition::resolver::cache: partition cache miss table_id=1 partition_key=1970-01-01
 2023-07-21T09:13:22.571795Z DEBUG ingester::buffer_tree::partition::resolver::catalog: upserting partition in catalog partition_key=1970-01-01 table_id=1 table=<unresolved>
 2023-07-21T09:13:22.604643Z DEBUG ingester::server::grpc::query: received query predicate predicate=Predicate { field_columns: [], range: None, exprs: [], value_expr: [] }
 2023-07-21T09:13:22.606391Z DEBUG iox_query::pruning: No expression on predicate
 2023-07-21T09:13:22.607344Z DEBUG ingester::query::result_instrumentation: completed streaming query results duration=Some(2.332469ms) row_count=2 record_batch_count=1 partition_count=1
 2023-07-21T09:13:22.656095Z DEBUG ingester::server::grpc::query: received query predicate predicate=Predicate { field_columns: [], range: None, exprs: [], value_expr: [] }
 2023-07-21T09:13:22.657013Z DEBUG iox_query::pruning: No expression on predicate
 2023-07-21T09:13:22.657851Z DEBUG ingester::query::result_instrumentation: completed streaming query results duration=Some(1.528981ms) row_count=2 record_batch_count=1 partition_count=1
 2023-07-21T09:13:22.706266Z  INFO ioxd_common: Received SIGTERM
 2023-07-21T09:13:22.706324Z  INFO ioxd_common: shutdown requested server_type=Ingester
 2023-07-21T09:13:22.706380Z  INFO ingester::init::graceful_shutdown: gracefully stopping ingester
 2023-07-21T09:13:22.706393Z  INFO ingester::init::graceful_shutdown: persisting all data before shutdown
 2023-07-21T09:13:22.706692Z DEBUG ingester::buffer_tree::partition: marking partition as persisting namespace_id=1 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 batch_ident=1
 2023-07-21T09:13:22.706804Z DEBUG ingester::persist::drain_buffer: read data for persistence partition_id=1 lock_wait=371.981µs
 2023-07-21T09:13:22.706879Z DEBUG ingester::persist::handle: enqueuing persistence task partition_id=1
 2023-07-21T09:13:22.706909Z DEBUG ingester::persist::handle: enqueue persist job to assigned worker partition_id=1
 2023-07-21T09:13:22.706946Z DEBUG ingester::persist::drain_buffer: queued all non-empty partitions for persist n_partitions=1
 2023-07-21T09:13:22.707007Z DEBUG ingester::persist::worker: compacting partition namespace_id=1 namespace_name=<unresolved> table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 sort_key=None
 2023-07-21T09:13:22.707182Z DEBUG schema::sort: computed sort key primary_key=["tag1", "tag2", "time"] sort_key=SortKey { columns: {"tag1": SortOptions { descending: false, nulls_first: true }, "tag2": SortOptions { descending: false, nulls_first: true }, "time": SortOptions { descending: false, nulls_first: true }} }
 2023-07-21T09:13:22.708149Z DEBUG iox_query::frontend::reorg: created compact plan for table table_name="mytable" plan=Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST [tag1:Dictionary(Int32, Utf8);N, tag2:Dictionary(Int32, Utf8);N, time:Timestamp(Nanosecond, None), val:Int64;N]
  TableScan: mytable [tag1:Dictionary(Int32, Utf8);N, tag2:Dictionary(Int32, Utf8);N, time:Timestamp(Nanosecond, None), val:Int64;N]
 2023-07-21T09:13:22.708236Z DEBUG iox_query::exec::context: create_physical_plan: initial plan text=Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST [tag1:Dictionary(Int32, Utf8);N, tag2:Dictionary(Int32, Utf8);N, time:Timestamp(Nanosecond, None), val:Int64;N]
  TableScan: mytable [tag1:Dictionary(Int32, Utf8);N, tag2:Dictionary(Int32, Utf8);N, time:Timestamp(Nanosecond, None), val:Int64;N]
 2023-07-21T09:13:22.708415Z DEBUG datafusion_optimizer::utils: inline_table_scan:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.708508Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.708601Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.708628Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.708651Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.708743Z DEBUG datafusion_optimizer::utils: type_coercion:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.708805Z DEBUG datafusion_optimizer::utils: count_wildcard_rule:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.708849Z DEBUG datafusion_optimizer::utils: Final analyzed plan:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.708870Z DEBUG datafusion_optimizer::analyzer: Analyzer took 0 ms    
 2023-07-21T09:13:22.708925Z DEBUG datafusion_optimizer::utils: Optimizer input (pass 0):
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709111Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709151Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.709190Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.709214Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.709239Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.709316Z DEBUG datafusion_optimizer::utils: unwrap_cast_in_comparison:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709347Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'replace_distinct_aggregate' (pass 0)    
 2023-07-21T09:13:22.709360Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_join' (pass 0)    
 2023-07-21T09:13:22.709371Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'decorrelate_predicate_subquery' (pass 0)    
 2023-07-21T09:13:22.709382Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'scalar_subquery_to_join' (pass 0)    
 2023-07-21T09:13:22.709392Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'extract_equijoin_predicate' (pass 0)    
 2023-07-21T09:13:22.709529Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709571Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'merge_projection' (pass 0)    
 2023-07-21T09:13:22.709584Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'rewrite_disjunctive_predicate' (pass 0)    
 2023-07-21T09:13:22.709634Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_duplicated_expr' (pass 0)    
 2023-07-21T09:13:22.709646Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_filter' (pass 0)    
 2023-07-21T09:13:22.709659Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_cross_join' (pass 0)    
 2023-07-21T09:13:22.709750Z DEBUG datafusion_optimizer::utils: common_sub_expression_eliminate:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709778Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_limit' (pass 0)    
 2023-07-21T09:13:22.709790Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'propagate_empty_relation' (pass 0)    
 2023-07-21T09:13:22.709801Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'filter_null_join_keys' (pass 0)    
 2023-07-21T09:13:22.709817Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_outer_join' (pass 0)    
 2023-07-21T09:13:22.709829Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_limit' (pass 0)    
 2023-07-21T09:13:22.709839Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_filter' (pass 0)    
 2023-07-21T09:13:22.709849Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'single_distinct_aggregation_to_group_by' (pass 0)    
 2023-07-21T09:13:22.709943Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.709978Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710015Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710039Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710062Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710134Z DEBUG datafusion_optimizer::utils: unwrap_cast_in_comparison:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.710203Z DEBUG datafusion_optimizer::utils: common_sub_expression_eliminate:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable
    
 2023-07-21T09:13:22.710299Z DEBUG datafusion_optimizer::utils: push_down_projection:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.710348Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_projection' (pass 0)    
 2023-07-21T09:13:22.710360Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_limit' (pass 0)    
 2023-07-21T09:13:22.710388Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710421Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710447Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710469Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710510Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710535Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710558Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710580Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710642Z DEBUG datafusion_optimizer::utils: influx_regex_to_datafusion_regex:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.710694Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'handle_gap_fill' (pass 0)    
 2023-07-21T09:13:22.710711Z DEBUG datafusion_optimizer::utils: Optimized plan (pass 0):
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.710768Z DEBUG datafusion_optimizer::utils: Optimizer input (pass 1):
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.710881Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.710918Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710961Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.710985Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711014Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711088Z DEBUG datafusion_optimizer::utils: unwrap_cast_in_comparison:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.711135Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'replace_distinct_aggregate' (pass 1)    
 2023-07-21T09:13:22.711147Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_join' (pass 1)    
 2023-07-21T09:13:22.711158Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'decorrelate_predicate_subquery' (pass 1)    
 2023-07-21T09:13:22.711168Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'scalar_subquery_to_join' (pass 1)    
 2023-07-21T09:13:22.711194Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'extract_equijoin_predicate' (pass 1)    
 2023-07-21T09:13:22.711299Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.711339Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'merge_projection' (pass 1)    
 2023-07-21T09:13:22.711352Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'rewrite_disjunctive_predicate' (pass 1)    
 2023-07-21T09:13:22.711402Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_duplicated_expr' (pass 1)    
 2023-07-21T09:13:22.711413Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_filter' (pass 1)    
 2023-07-21T09:13:22.711441Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_cross_join' (pass 1)    
 2023-07-21T09:13:22.711523Z DEBUG datafusion_optimizer::utils: common_sub_expression_eliminate:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.711593Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_limit' (pass 1)    
 2023-07-21T09:13:22.711610Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'propagate_empty_relation' (pass 1)    
 2023-07-21T09:13:22.711621Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'filter_null_join_keys' (pass 1)    
 2023-07-21T09:13:22.711631Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_outer_join' (pass 1)    
 2023-07-21T09:13:22.711649Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_limit' (pass 1)    
 2023-07-21T09:13:22.711660Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_filter' (pass 1)    
 2023-07-21T09:13:22.711670Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'single_distinct_aggregation_to_group_by' (pass 1)    
 2023-07-21T09:13:22.711775Z DEBUG datafusion_optimizer::utils: simplify_expressions:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.711825Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711863Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711887Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711909Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.711989Z DEBUG datafusion_optimizer::utils: unwrap_cast_in_comparison:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.712068Z DEBUG datafusion_optimizer::utils: common_sub_expression_eliminate:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.712106Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_projection' (pass 1)    
 2023-07-21T09:13:22.712123Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'eliminate_projection' (pass 1)    
 2023-07-21T09:13:22.712134Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'push_down_limit' (pass 1)    
 2023-07-21T09:13:22.712151Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712183Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712214Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712237Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712274Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712298Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712323Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712345Z DEBUG sqlparser::parser: Parsing sql 'mytable'...    
 2023-07-21T09:13:22.712398Z DEBUG datafusion_optimizer::utils: influx_regex_to_datafusion_regex:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.712449Z DEBUG datafusion_optimizer::optimizer: Plan unchanged by optimizer rule 'handle_gap_fill' (pass 1)    
 2023-07-21T09:13:22.712466Z DEBUG datafusion_optimizer::utils: Optimized plan (pass 1):
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.712523Z DEBUG datafusion_optimizer::optimizer: optimizer pass 1 did not make changes    
 2023-07-21T09:13:22.712533Z DEBUG datafusion_optimizer::utils: Final optimized plan:
 Sort: mytable.tag1 ASC NULLS FIRST, mytable.tag2 ASC NULLS FIRST, mytable.time ASC NULLS FIRST
  TableScan: mytable projection=[tag1, tag2, time, val]
    
 2023-07-21T09:13:22.712554Z DEBUG datafusion_optimizer::optimizer: Optimizer took 3 ms    
 2023-07-21T09:13:22.713122Z DEBUG datafusion::physical_planner: Input physical plan:
 SortExec: expr=[tag1@0 ASC,tag2@1 ASC,time@2 ASC]
  ProjectionExec: expr=[tag1@0 as tag1, tag2@1 as tag2, time@2 as time, val@3 as val]
    DeduplicateExec: [tag1@0 ASC,tag2@1 ASC,time@2 ASC]
      UnionExec
        RecordBatchesExec: batches_groups=1 batches=1 total_rows=2

    
 2023-07-21T09:13:22.716171Z DEBUG datafusion::physical_planner: Optimized physical plan:
 ProjectionExec: expr=[tag1@1 as tag1, tag2@2 as tag2, time@3 as time, val@4 as val]
  DeduplicateExec: [tag1@1 ASC,tag2@2 ASC,time@3 ASC]
    SortExec: expr=[tag1@1 ASC,tag2@2 ASC,time@3 ASC,__chunk_order@0 ASC]
      RecordBatchesExec: batches_groups=1 batches=1 total_rows=2

    
 2023-07-21T09:13:22.716383Z DEBUG iox_query::exec::context: create_physical_plan: plan to run text=ProjectionExec: expr=[tag1@1 as tag1, tag2@2 as tag2, time@3 as time, val@4 as val]
  DeduplicateExec: [tag1@1 ASC,tag2@2 ASC,time@3 ASC]
    SortExec: expr=[tag1@1 ASC,tag2@2 ASC,time@3 ASC,__chunk_order@0 ASC]
      RecordBatchesExec: batches_groups=1 batches=1 total_rows=2

 2023-07-21T09:13:22.719916Z DEBUG iox_query::provider::deduplicate: End building stream for DeduplicationExec::execute partition=0
 2023-07-21T09:13:22.720388Z DEBUG ingester::persist::worker: uploading partition parquet namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 sort_key=tag1, tag2, time,
 2023-07-21T09:13:22.720521Z DEBUG parquet_file::serialize: IOxMetaData provided for serializing the data into the in-memory buffer meta=IoxMetadata { object_store_id: 20c6f624-6d6f-47f6-97cb-49298cc8b903, creation_timestamp: 2023-07-21T09:13:22.720498832+00:00, namespace_id: NamespaceId(1), namespace_name: "6578172476526892_6900599325749272", table_id: TableId(1), table_name: "mytable", partition_key: PartitionKey("1970-01-01"), compaction_level: Initial, sort_key: Some(SortKey { columns: {"tag1": SortOptions { descending: false, nulls_first: true }, "tag2": SortOptions { descending: false, nulls_first: true }, "time": SortOptions { descending: false, nulls_first: true }} }), max_l0_created_at: 2023-07-21T09:13:22.720498832+00:00 }
 2023-07-21T09:13:22.721005Z DEBUG iox_query::provider::deduplicate: before sending the left over batch
 2023-07-21T09:13:22.725874Z DEBUG iox_query::provider::deduplicate: done sending the left over batch
 2023-07-21T09:13:22.727825Z DEBUG parquet_file::serialize: Created parquet file num_batches=2 num_rows=2 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 write_batch_size=1024 max_row_group_size=1048576
 2023-07-21T09:13:22.729032Z DEBUG parquet_file::storage: Uploading parquet to object store file_size=2260 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 total_time_to_create_parquet_bytes=8.514726ms
 2023-07-21T09:13:22.729675Z DEBUG ingester::persist::worker: partition parquet uploaded namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 file_size=2260
 2023-07-21T09:13:22.732051Z DEBUG ingester::persist::worker: updating partition sort key object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 new_sort_key=SortKey { columns: {"tag1": SortOptions { descending: false, nulls_first: true }, "tag2": SortOptions { descending: false, nulls_first: true }, "time": SortOptions { descending: false, nulls_first: true }} } old_sort_key=None
 2023-07-21T09:13:22.741358Z DEBUG iox_catalog::postgres: partition sort key cas successful partition_id=Deterministic(PartitionHashId([184, 98, 167, 233, 179, 41, 238, 106, 65, 140, 222, 25, 17, 152, 234, 235, 21, 18, 117, 63, 25, 184, 122, 129, 222, 242, 174, 108, 61, 14, 210, 55])) old_sort_key=[] new_sort_key=["tag1", "tag2", "time"]
 2023-07-21T09:13:22.741554Z DEBUG ingester::persist::worker: adjusted partition sort key object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 old_sort_key=None new_sort_key=tag1, tag2, time,
 2023-07-21T09:13:22.741619Z DEBUG ingester::persist::worker: updating catalog parquet table namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 parquet_table_data=ParquetFileParams { namespace_id: NamespaceId(1), table_id: TableId(1), partition_id: PartitionId(1), partition_hash_id: Some(PartitionHashId([184, 98, 167, 233, 179, 41, 238, 106, 65, 140, 222, 25, 17, 152, 234, 235, 21, 18, 117, 63, 25, 184, 122, 129, 222, 242, 174, 108, 61, 14, 210, 55])), object_store_id: 20c6f624-6d6f-47f6-97cb-49298cc8b903, min_time: Timestamp(123456), max_time: Timestamp(1234567), file_size_bytes: 2260, row_count: 2, compaction_level: Initial, created_at: Timestamp(1689930802720498832), column_set: ColumnSet([ColumnId(1), ColumnId(2), ColumnId(3), ColumnId(4)]), max_l0_created_at: Timestamp(1689930802720498832) }
 2023-07-21T09:13:22.747714Z DEBUG ingester::persist::worker: parquet file added to catalog namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 parquet_table_data=ParquetFileParams { namespace_id: NamespaceId(1), table_id: TableId(1), partition_id: PartitionId(1), partition_hash_id: Some(PartitionHashId([184, 98, 167, 233, 179, 41, 238, 106, 65, 140, 222, 25, 17, 152, 234, 235, 21, 18, 117, 63, 25, 184, 122, 129, 222, 242, 174, 108, 61, 14, 210, 55])), object_store_id: 20c6f624-6d6f-47f6-97cb-49298cc8b903, min_time: Timestamp(123456), max_time: Timestamp(1234567), file_size_bytes: 2260, row_count: 2, compaction_level: Initial, created_at: Timestamp(1689930802720498832), column_set: ColumnSet([ColumnId(1), ColumnId(2), ColumnId(3), ColumnId(4)]), max_l0_created_at: Timestamp(1689930802720498832) } parquet_file_id=ParquetFileId(1)
 2023-07-21T09:13:22.747846Z DEBUG ingester::buffer_tree::partition: marking partition persistence complete batch_ident=1 persistence_count=1 namespace_id=1 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 batch_ident=1
 2023-07-21T09:13:22.747974Z  INFO ingester::persist::context: persisted partition object_store_id=20c6f624-6d6f-47f6-97cb-49298cc8b903 namespace_id=1 namespace_name=6578172476526892_6900599325749272 table_id=1 table=mytable partition_id=1 partition_key=1970-01-01 total_persist_duration=41.073315ms active_persist_duration=40.985491ms queued_persist_duration=87.824µs n_writes=2
 2023-07-21T09:13:22.748714Z DEBUG ingester::wal::reference_tracker::actor: notified of persisted data namespace_id=1 table_id=1 partition_id=1 sequence_number_set=SequenceNumberSet(Treemap<[1, 2]>)
 2023-07-21T09:13:22.748825Z DEBUG ingester::wal::reference_tracker::actor: retaining file-less IDs n=2
 2023-07-21T09:13:22.748858Z DEBUG ingester::wal::reference_tracker::actor: updated reference state num_files=0 files=[] persisted_set_len=2
 2023-07-21T09:13:22.749822Z  INFO ingester::init::graceful_shutdown: persisted all data - stopping ingester
 2023-07-21T09:13:22.749936Z DEBUG hyper::server::shutdown: signal received, starting graceful shutdown
 2023-07-21T09:13:22.750015Z  INFO ioxd_common: HTTP server shutdown server_type=Ingester
 2023-07-21T09:13:22.750040Z  INFO ioxd_common: frontend shutdown completed server_type=Ingester
 2023-07-21T09:13:22.750082Z  INFO ioxd_common: backend shutdown completed server_type=Ingester
 2023-07-21T09:13:22.750171Z  INFO influxdb_iox::commands::run::main: done serving, draining futures grpc_bind_address=SocketAddr(127.0.0.1:8190) http_bind_address=Some(SocketAddr(127.0.0.1:8191)) server_type=Ingester
 2023-07-21T09:13:22.750265Z DEBUG influxdb_iox::commands::run::main: handle returned server_type=Ingester
 2023-07-21T09:13:22.750484Z DEBUG wal::writer_thread: stopping WAL IO thread
 2023-07-21T09:13:22.750482Z  WARN executor: DedicatedExecutor dropped without calling shutdown()
 2023-07-21T09:13:22.750706Z  WARN executor: DedicatedExecutor dropped without waiting for worker termination
 2023-07-21T09:13:22.765513Z  WARN executor: DedicatedExecutor dropped without calling shutdown()
 2023-07-21T09:13:22.765638Z  WARN executor: DedicatedExecutor dropped without waiting for worker termination
 2023-07-21T09:13:23.212214Z  INFO clap_blocks::catalog_dsn: Catalog: Postgres postgres_schema_name=XDPhlKuZlNvCkAksheDD
 2023-07-21T09:13:23.231055Z  INFO iox_catalog::postgres: connected to config store application_name=ingester
 2023-07-21T09:13:23.231857Z DEBUG datafusion_execution::memory_pool::pool: Created new GreedyMemoryPool(pool_size=18446744073709551615)    
 2023-07-21T09:13:23.231950Z  INFO clap_blocks::object_store: Object Store db_dir="/tmp/.tmpPMx6pk" object_store_type="Directory"
 2023-07-21T09:13:23.235790Z  INFO wal: Initalizing Write Ahead Log (WAL) wal_dir="/tmp/.tmpsjT3o2"
 2023-07-21T09:13:23.258086Z  INFO ingester::persist::handle: initialised persist task n_workers=5 persist_queue_depth=250
 2023-07-21T09:13:23.259343Z  INFO ingester::init::wal_replay: found wal files for replay n_files=2
 2023-07-21T09:13:23.259454Z  INFO ingester::init::wal_replay: replaying wal file file_number=1 n_files=2 file_id=2 size=318
 thread 'main' panicked at 'attempt to apply unsequenced wal op', /home/rust/project/ingester/src/init/wal_replay.rs:245:22
 stack backtrace:
   0: rust_begin_unwind
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:67:14
   2: core::panicking::panic_display
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:150:5
   3: core::panicking::panic_str
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:134:5
   4: core::option::expect_failed
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/option.rs:1932:5
   5: core::option::Option<T>::expect
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/option.rs:898:21
   6: ingester::init::wal_replay::replay_file::{{closure}}
             at /home/rust/project/ingester/src/init/wal_replay.rs:244:42
   7: ingester::init::wal_replay::replay::{{closure}}
             at /home/rust/project/ingester/src/init/wal_replay.rs:115:59
   8: ingester::init::new::{{closure}}
             at /home/rust/project/ingester/src/init.rs:363:14
   9: ioxd_ingester::create_ingester_server_type::{{closure}}
             at /home/rust/project/ioxd_ingester/src/lib.rs:226:6
  10: influxdb_iox::commands::run::ingester::command::{{closure}}
             at ./src/commands/run/ingester.rs:127:6
  11: influxdb_iox::commands::run::command::{{closure}}
             at ./src/commands/run/mod.rs:105:70
  12: influxdb_iox::main::{{closure}}
             at ./src/main.rs:314:65
  13: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  22: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/runtime.rs:313:45
  23: influxdb_iox::main
             at ./src/main.rs:236:5
  24: core::ops::function::FnOnce::call_once
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:250:5
 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
 2023-07-21T09:13:33.156289Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.156295Z  INFO test_helpers_end_to_end: End Ingester Output
 2023-07-21T09:13:33.156301Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.200724Z  INFO test_helpers_end_to_end::server_fixture: Killed child politely
 2023-07-21T09:13:33.200779Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.200788Z  INFO test_helpers_end_to_end: Start Router Output
 2023-07-21T09:13:33.200794Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:20.900166Z  INFO clap_blocks::catalog_dsn: Catalog: Postgres postgres_schema_name=XDPhlKuZlNvCkAksheDD
 2023-07-21T09:13:20.906415Z  INFO iox_catalog::postgres: connected to config store application_name=router
 2023-07-21T09:13:20.906597Z  INFO clap_blocks::object_store: Object Store db_dir="/tmp/.tmpPMx6pk" object_store_type="Directory"
 2023-07-21T09:13:20.907999Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8190
 2023-07-21T09:13:20.909679Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8190
 2023-07-21T09:13:20.909794Z DEBUG router::dml_handlers::rpc_write: write replication factor n_copies=1
 2023-07-21T09:13:20.911239Z  INFO router::dml_handlers::rpc_write::lazy_connector: connected to upstream ingester endpoint=http://127.0.0.1:8190/
 2023-07-21T09:13:20.932540Z  INFO influxdb_iox::commands::run::router: starting router
 2023-07-21T09:13:20.935381Z  INFO influxdb_iox::commands::run::main: InfluxDB IOx server starting git_hash=e67d138dd609642a8070c831bddd6b80ca6563b5 version=0.1.0 uuid=87e12559-635c-41b0-82b1-8869632a02a5 num_cpus=16 build_malloc_conf=�
 2023-07-21T09:13:20.935733Z  INFO influxdb_iox::commands::run::main: Binding gRPC services grpc_bind_address=SocketAddr(127.0.0.1:8212) server_type=Router
 2023-07-21T09:13:20.935904Z  INFO ioxd_common: bound gRPC listener local_addr=127.0.0.1:8212
 2023-07-21T09:13:20.935932Z  INFO influxdb_iox::commands::run::main: Completed bind of gRPC, binding http http_bind_address=SocketAddr(127.0.0.1:8211) server_type=Router
 2023-07-21T09:13:20.935981Z  INFO ioxd_common: bound HTTP listener bind_addr=127.0.0.1:8211
 2023-07-21T09:13:20.936018Z  INFO ioxd_common: gRPC server listening server_type=Router
 2023-07-21T09:13:20.936042Z  INFO ioxd_common: InfluxDB IOx Router server ready    
 2023-07-21T09:13:20.963303Z DEBUG influxdb_iox::commands::run::main: wait for handle server_type=Router
 2023-07-21T09:13:20.969341Z  INFO ioxd_common: HTTP server listening server_type=Router
 2023-07-21T09:13:20.969714Z DEBUG ioxd_common::http: Processing request request=Request { method: GET, uri: /health, version: HTTP/1.1, headers: {"accept": "*/*", "host": "127.0.0.1:8211"}, body: Body(Empty) }
 2023-07-21T09:13:20.969765Z DEBUG ioxd_common::http: Successfully processed request response=Response { status: 200, version: HTTP/1.1, headers: {}, body: Body(Full(b"OK")) }
 2023-07-21T09:13:22.520608Z DEBUG ioxd_common::http: Processing request request=Request { method: POST, uri: /api/v2/write?org=6578172476526892&bucket=6900599325749272, version: HTTP/1.1, headers: {"host": "127.0.0.1:8211", "content-length": "36"}, body: Body(Streaming) }
 2023-07-21T09:13:22.521209Z DEBUG router::server::http: routing write num_lines=1 num_fields=1 num_tables=1 precision=Nanoseconds body_size=36 namespace=6578172476526892_6900599325749272 duration=360.431µs
 2023-07-21T09:13:22.524619Z  WARN router::namespace_cache::read_through_cache: failed to retrieve namespace schema error=namespace 6578172476526892_6900599325749272 not found namespace=6578172476526892_6900599325749272
 2023-07-21T09:13:22.527343Z DEBUG router::namespace_resolver::ns_autocreation: created namespace namespace=6578172476526892_6900599325749272
 2023-07-21T09:13:22.569114Z DEBUG tower::buffer::worker: service.ready=true processing request
 2023-07-21T09:13:22.578014Z DEBUG router::dml_handlers::rpc_write: dispatched write to ingester partition_key=1970-01-01 table_count=1 namespace=6578172476526892_6900599325749272 namespace_id=1 approx_size=1907
 2023-07-21T09:13:22.578208Z DEBUG ioxd_common::http: Successfully processed request response=Response { status: 204, version: HTTP/1.1, headers: {}, body: Body(Empty) }
 2023-07-21T09:13:22.751000Z DEBUG hyper::client::service: connection error: hyper::Error(Io, Custom { kind: BrokenPipe, error: "connection closed because of a broken pipe" })
 2023-07-21T09:13:33.172934Z  INFO ioxd_common: Received SIGTERM
 2023-07-21T09:13:33.173613Z  INFO ioxd_common: shutdown requested server_type=Router
 2023-07-21T09:13:33.173800Z DEBUG hyper::server::shutdown: signal received, starting graceful shutdown
 2023-07-21T09:13:33.174193Z  INFO ioxd_common: frontend shutdown completed server_type=Router
 2023-07-21T09:13:33.174228Z  INFO ioxd_common: backend shutdown completed server_type=Router
 2023-07-21T09:13:33.174368Z  INFO influxdb_iox::commands::run::main: done serving, draining futures grpc_bind_address=SocketAddr(127.0.0.1:8212) http_bind_address=Some(SocketAddr(127.0.0.1:8211)) server_type=Router
 2023-07-21T09:13:33.174608Z DEBUG influxdb_iox::commands::run::main: handle returned server_type=Router
 2023-07-21T09:13:33.174819Z DEBUG tower::buffer::worker: buffer closing; waking pending tasks
 2023-07-21T09:13:33.200824Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.200831Z  INFO test_helpers_end_to_end: End Router Output
 2023-07-21T09:13:33.200836Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.225508Z  INFO test_helpers_end_to_end::server_fixture: Killed child politely
 2023-07-21T09:13:33.225552Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.225561Z  INFO test_helpers_end_to_end: Start Querier Output
 2023-07-21T09:13:33.225567Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:22.408400Z  INFO clap_blocks::catalog_dsn: Catalog: Postgres postgres_schema_name=XDPhlKuZlNvCkAksheDD
 2023-07-21T09:13:22.423221Z  INFO iox_catalog::postgres: connected to config store application_name=querier
 2023-07-21T09:13:22.423333Z  INFO clap_blocks::object_store: Object Store db_dir="/tmp/.tmpPMx6pk" object_store_type="Directory"
 2023-07-21T09:13:22.423659Z  INFO influxdb_iox::commands::run::querier: using specified number of threads per thread pool num_threads=4
 2023-07-21T09:13:22.423692Z  INFO influxdb_iox::commands::run::querier: using ingester addresses ingester_addresses=[IngesterAddress { uri: http://127.0.0.1:8190/ }]
 2023-07-21T09:13:22.424423Z DEBUG datafusion_execution::memory_pool::pool: Created new GreedyMemoryPool(pool_size=8589934592)    
 2023-07-21T09:13:22.428021Z  INFO influxdb_iox::commands::run::querier: starting querier
 2023-07-21T09:13:22.438943Z  INFO influxdb_iox::commands::run::main: InfluxDB IOx server starting git_hash=e67d138dd609642a8070c831bddd6b80ca6563b5 version=0.1.0 uuid=f3444212-6cf1-4175-a12e-494ff344403e num_cpus=16 build_malloc_conf=�
 2023-07-21T09:13:22.439206Z DEBUG influxdb_iox::commands::run::main: wait for handle server_type=Querier
 2023-07-21T09:13:22.441587Z  INFO influxdb_iox::commands::run::main: Binding gRPC services grpc_bind_address=SocketAddr(127.0.0.1:8239) server_type=Querier
 2023-07-21T09:13:22.448807Z  INFO ioxd_common: bound gRPC listener local_addr=127.0.0.1:8239
 2023-07-21T09:13:22.448843Z  INFO influxdb_iox::commands::run::main: Completed bind of gRPC, binding http http_bind_address=SocketAddr(127.0.0.1:8238) server_type=Querier
 2023-07-21T09:13:22.448910Z  INFO ioxd_common: bound HTTP listener bind_addr=127.0.0.1:8238
 2023-07-21T09:13:22.448960Z  INFO ioxd_common: gRPC server listening server_type=Querier
 2023-07-21T09:13:22.448987Z  INFO ioxd_common: InfluxDB IOx Querier server ready    
 2023-07-21T09:13:22.472969Z  INFO ioxd_common: HTTP server listening server_type=Querier
 2023-07-21T09:13:22.474104Z DEBUG ioxd_common::http: Processing request request=Request { method: GET, uri: /health, version: HTTP/1.1, headers: {"accept": "*/*", "host": "127.0.0.1:8238"}, body: Body(Empty) }
 2023-07-21T09:13:22.474260Z DEBUG ioxd_common::http: Successfully processed request response=Response { status: 200, version: HTTP/1.1, headers: {}, body: Body(Full(b"OK")) }
 2023-07-21T09:13:33.201311Z  INFO ioxd_common: Received SIGTERM
 2023-07-21T09:13:33.201381Z  INFO ioxd_common: shutdown requested server_type=Querier
 2023-07-21T09:13:33.201479Z DEBUG hyper::server::shutdown: signal received, starting graceful shutdown
 2023-07-21T09:13:33.201680Z DEBUG hyper::server::shutdown: signal received, starting graceful shutdown
 2023-07-21T09:13:33.201748Z  INFO ioxd_common: HTTP server shutdown server_type=Querier
 2023-07-21T09:13:33.201776Z  INFO ioxd_common: frontend shutdown completed server_type=Querier
 2023-07-21T09:13:33.207650Z  INFO ioxd_common: backend shutdown completed server_type=Querier
 2023-07-21T09:13:33.207809Z  INFO influxdb_iox::commands::run::main: done serving, draining futures grpc_bind_address=SocketAddr(127.0.0.1:8239) http_bind_address=Some(SocketAddr(127.0.0.1:8238)) server_type=Querier
 2023-07-21T09:13:33.208029Z DEBUG influxdb_iox::commands::run::main: handle returned server_type=Querier
 2023-07-21T09:13:33.225596Z  INFO test_helpers_end_to_end: ****************
 2023-07-21T09:13:33.225602Z  INFO test_helpers_end_to_end: End Querier Output
 2023-07-21T09:13:33.225608Z  INFO test_helpers_end_to_end: ****************


 failures:
    end_to_end_cases::ingester::ingester_flight_api

 test result: FAILED. 294 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 39.75s

 error: test failed, to rerun pass `--test end_to_end`