Software · head to head
Dask vs BigQuery ML
The short version
- Each has a real cost: Dask each Dask task carries between 200 microseconds and 1 millisecond of scheduler overhead, so graphs of millions of tasks add 10 minutes to hours of pure overhead; BigQuery ML not available in BigQuery's Standard edition, so the cheapest tier cannot use it
- They diverge on capability: Dask covers Parallel computing, BigQuery ML covers SQL-based ML.
Where they differ
Only the attributes on which Dask and BigQuery ML actually diverge.
| Attribute | Dask | BigQuery ML |
|---|---|---|
| Pricing model | open-source | usage-based |
| Platforms | Linux, Mac, Windows | Web |
| Founded | 2015 | 2008 |
Identical on both: starting price (Free), free tier (Yes), user rating (Not yet rated), category (Unknown).
What each one covers
Drawn from each product's published feature list. An absence here means we hold no record of it - not that the product lacks it.
Only in Dask
- Parallel computing
- Distributed DataFrames
- Lazy evaluation
- Dynamic task scheduling
- Dashboard
- NumPy
- Pandas
- scikit-learn
Only in BigQuery ML
- SQL-based ML
- AutoML Tables
- Model export
- Prediction functions
- Feature preprocessing
- BigQuery
- Vertex AI
- TensorFlow
What people use each for
The jobs each tool is most often brought in to do.
Dask
- Scaling pandas and NumPy workloads beyond a single machine's memorynot BigQuery ML
- Parallelising custom Python task graphsnot BigQuery ML
- Processing larger than memory arrays and dataframes on a clusternot BigQuery ML
BigQuery ML
- Training models in SQL without exporting datanot Dask
- Linear and logistic regression on warehouse datanot Dask
- K-means clustering and matrix factorisation for recommendationsnot Dask
- Time series forecasting with ARIMA_PLUSnot Dask
- Running imported ONNX, TensorFlow or XGBoost models against BigQuery datanot Dask
Where each one falls short
Documented limitations, not opinions. Every one is a constraint you would hit in normal use.
Dask
- Each Dask task carries between 200 microseconds and 1 millisecond of scheduler overhead, so graphs of millions of tasks add 10 minutes to hours of pure overhead
- Partition sizing is left to the user: chunks must fit several times over in worker memory, and both oversized and undersized chunks are documented failure modes
- Embedding large locally created DataFrames or Arrays into a Dask computation is documented as a practice to avoid because of network overhead
- Calling compute repeatedly in a loop rather than batching prevents parallelisation of queries
- The documentation itself advises trying better algorithms, file formats or sampling before adopting Dask
BigQuery ML
- Not available in BigQuery's Standard edition, so the cheapest tier cannot use it
- Billed through BigQuery compute and storage rather than as its own product, so training cost tracks data scanned
- Remote models incur extra Agent Platform charges on top
- Externally trained model types such as boosted trees and AutoML run through Agent Platform rather than inside BigQuery
Pricing, plan by plan
Dask
Free- Open SourceFree
- Parallel computing
- Distributed DataFrames
- ML integration
BigQuery ML
Free- Free TierFree
- 10GB storage
- 1TB queries
- On-Demand$5/TB
- Pay per TB scanned
- ML training costs
Which should you pick?
Choose Dask if
- You need parallel computing.
- You want to start without paying.
- You work on Linux, Mac, Windows.
- You also want distributed dataframes.
Choose BigQuery ML if
- You need sql-based ml.
- You want to start without paying.
- You also want automl tables.
Questions people ask
- Is Dask or BigQuery ML better?
- Neither clearly leads. Dask starts at Free and BigQuery ML at Free, and user ratings are close enough to be indistinguishable. Choose on capability and platform support.
- Which is cheaper, Dask or BigQuery ML?
- Dask starts at Free and BigQuery ML at Free.
- Does Dask or BigQuery ML run on more platforms?
- Dask runs on Linux, Mac, Windows. BigQuery ML runs on Web.
- Can I use Dask for free?
- Both have a free tier, so you can try either at no cost before committing.
- What is Dask best used for?
- Dask is most often used for scaling pandas and numpy workloads beyond a single machine's memory, parallelising custom python task graphs, processing larger than memory arrays and dataframes on a cluster. Of those, scaling pandas and numpy workloads beyond a single machine's memory and parallelising custom python task graphs are not what BigQuery ML is typically brought in for.
- What can Dask do that BigQuery ML cannot?
- Dask covers Parallel computing, Distributed DataFrames, Lazy evaluation, Dynamic task scheduling. BigQuery ML covers SQL-based ML, AutoML Tables, Model export, Prediction functions.