Metabase previous update: 18th of July 2022
Snowflake balance top up to 10,000$: Oct 2022
INCIDENT SUMMARY
Metabase got stuck on a sync loop, constantly querying Snowflake since 1st Jan 2023 at 00:00 when the year changed, and continued until I came back from holidays and was able to detect it. At the change of year, Metabase got stuck in this sync loop, repeatedly querying Snowflake for the same information, even though it had already been synced. This resulted in an excessive amount of Cloud Services usage that had a big cost for us.
This Metabase heavy syncing traffic gets mixed in the activity log with the normal querying and it becomes stealthy. What made it surface was the lack of querying activity through Metabase during the holidays, which made surface the constant non-stop sync starting on 2023-01-01
Since Sept 2022 (spikes start): $1354,74
Cost of Metabase’s
show /* JDBC:DatabaseMetaData.getColumns() */ columns in database "FIVETRAN”
and
show /* JDBC:DatabaseMetaData.getPrimaryKeys() */ primary keys in database "FIVETRAN”
This is non-productive data querying that slipped under the radar. These are just highly repeated columns/schemas querying.
These are the Cloud Services Compute costs that have been charged for exceeding 10% of the daily warehouse usage each day.
Metabase has been constantly triggering Cloud Services by asking and re-asking again, for the same sync information. Cloud Services are usually free as long as not exceeded 10% of daily use.
This Metabase querying has been exceeding by far this threshold.
1 - 12 Jan 2023: 177.16 credits = $662.58