Machine Learning & Data Science · head to head
BigQuery ML vs Dask
BigQuery ML
Machine Learning & Data Science
Machine learning in BigQuery using SQL
- From
- Free
- Rated
- -
The short version
- Each has a real cost: BigQuery ML not available in BigQuery's Standard edition, so the cheapest tier cannot use it; Dask each Dask task carries between 200 microseconds and 1 millisecond of scheduler overhead, so graphs of millions of tasks add 10 minutes to hours of pure overhead
- They diverge on capability: BigQuery ML covers SQL-based ML, Dask covers Parallel computing.
Where they differ
Only the attributes on which BigQuery ML and Dask actually diverge.
| Attribute | BigQuery ML | Dask |
|---|---|---|
| Pricing model | usage-based | open-source |
| Platforms | Web | Linux, Mac, Windows |
| Founded | 2008 | 2015 |
Identical on both: starting price (Free), free tier (Yes), user rating (Not yet rated), category (Machine Learning & Data Science).
What each one covers
Drawn from each product's published feature list. An absence here means we hold no record of it - not that the product lacks it.
Only in BigQuery ML
- SQL-based ML
- AutoML Tables
- Model export
- Prediction functions
- Feature preprocessing
- BigQuery
- Vertex AI
- TensorFlow
Only in Dask
- Parallel computing
- Distributed DataFrames
- Lazy evaluation
- Dynamic task scheduling
- Dashboard
- NumPy
- Pandas
- scikit-learn
What people use each for
The jobs each tool is most often brought in to do.
BigQuery ML
- Training models in SQL without exporting datanot Dask
- Linear and logistic regression on warehouse datanot Dask
- K-means clustering and matrix factorisation for recommendationsnot Dask
- Time series forecasting with ARIMA_PLUSnot Dask
- Running imported ONNX, TensorFlow or XGBoost models against BigQuery datanot Dask
Dask
- Scaling pandas and NumPy workloads beyond a single machine's memorynot BigQuery ML
- Parallelising custom Python task graphsnot BigQuery ML
- Processing larger than memory arrays and dataframes on a clusternot BigQuery ML
Where each one falls short
Documented limitations, not opinions. Every one is a constraint you would hit in normal use.
BigQuery ML
- Not available in BigQuery's Standard edition, so the cheapest tier cannot use it
- Billed through BigQuery compute and storage rather than as its own product, so training cost tracks data scanned
- Remote models incur extra Agent Platform charges on top
- Externally trained model types such as boosted trees and AutoML run through Agent Platform rather than inside BigQuery
Dask
- Each Dask task carries between 200 microseconds and 1 millisecond of scheduler overhead, so graphs of millions of tasks add 10 minutes to hours of pure overhead
- Partition sizing is left to the user: chunks must fit several times over in worker memory, and both oversized and undersized chunks are documented failure modes
- Embedding large locally created DataFrames or Arrays into a Dask computation is documented as a practice to avoid because of network overhead
- Calling compute repeatedly in a loop rather than batching prevents parallelisation of queries
- The documentation itself advises trying better algorithms, file formats or sampling before adopting Dask
Pricing, plan by plan
BigQuery ML
Free- Free TierFree
- 10GB storage
- 1TB queries
- On-Demand$5/TB
- Pay per TB scanned
- ML training costs
Dask
Free- Open SourceFree
- Parallel computing
- Distributed DataFrames
- ML integration
Which should you pick?
Choose BigQuery ML if
- You need sql-based ml.
- You want to start without paying.
- You also want automl tables.
Choose Dask if
- You need parallel computing.
- You want to start without paying.
- You work on Linux, Mac, Windows.
- You also want distributed dataframes.
Questions people ask
- Is BigQuery ML or Dask better?
- Neither clearly leads. BigQuery ML starts at Free and Dask at Free, and user ratings are close enough to be indistinguishable. Choose on capability and platform support.
- Which is cheaper, BigQuery ML or Dask?
- BigQuery ML starts at Free and Dask at Free.
- Does BigQuery ML or Dask run on more platforms?
- BigQuery ML runs on Web. Dask runs on Linux, Mac, Windows.
- Can I use BigQuery ML for free?
- Both have a free tier, so you can try either at no cost before committing.
- What is BigQuery ML best used for?
- BigQuery ML is most often used for training models in sql without exporting data, linear and logistic regression on warehouse data, k-means clustering and matrix factorisation for recommendations, time series forecasting with arima_plus. Of those, training models in sql without exporting data and linear and logistic regression on warehouse data are not what Dask is typically brought in for.
- What can BigQuery ML do that Dask cannot?
- BigQuery ML covers SQL-based ML, AutoML Tables, Model export, Prediction functions. Dask covers Parallel computing, Distributed DataFrames, Lazy evaluation, Dynamic task scheduling.