Conversation
| value -> String.to_float(value) | ||
| end | ||
|
|
||
| bq_compare_sample_ratio = |
There was a problem hiding this comment.
i think we don't need a separate env var for this, we can reuse the ingest_sample_ratio for sampling the insert mechanism
| defp compute_batch_attrs(messages, bq_api_tag) do | ||
| event_count = length(messages) | ||
| bytes = messages |> Enum.map(&:erlang.external_size(&1.data.body)) |> Enum.sum() | ||
|
|
||
| %{insert_method: bq_api_tag, batch_event_count: event_count, batch_bytes: bytes} | ||
| end |
There was a problem hiding this comment.
I'm assuming that the thought process for adding this attr in is to compare whether payload size plays a role in the performance? might be better to compute this lower down in the request payload construction if so, this is before any manipulations are performed and wouldn't be as informative.
| "/api/events" <> _ -> ingest_config() | ||
| "/endpoints/query" <> _ -> endpoint_config() | ||
| "/api/endpoints/query" <> _ -> endpoint_config() | ||
| case {span_name, Map.get(attributes, :"url.path")} do |
There was a problem hiding this comment.
we probably can check if the span name contains _pipeline_ or an ingest_ prefix that would clearly indicate that it is part of the ingestion pipeline.
| end | ||
|
|
||
| defp bq_compare_config do | ||
| prob = Application.get_env(:logflare, :bq_compare_sample_ratio) |
There was a problem hiding this comment.
as above, should use ingest sample ratio
2 participants
Closes O11Y-1580