Softwr
Apache Hadoop logo

Apache Hadoop

The original open source framework for distributed storage and batch processing on commodity servers, now largely a legacy platform.

As of 13 November 2026, Apache Hadoop is free to use. Hadoop provides HDFS for distributed storage and YARN for cluster scheduling, and still runs large on-premises estates. Softwr lists it under Databases.

Overview

What Apache Hadoop does

Apache Hadoop is an Apache Software Foundation project licensed under Apache 2.0, made up of four parts: HDFS, a distributed filesystem that replicates blocks across nodes; YARN, a cluster resource manager and scheduler; MapReduce, the original batch processing model; and Hadoop Common, the shared libraries. It is written in Java and designed to run on racks of commodity servers with directly attached disks, tolerating individual machine failures through replication. A large surrounding ecosystem grew around it, including Hive, HBase, Ranger, Oozie and Ozone, and Spark and Flink both still run under YARN. The distinguishing property today is not technical, it is economic and it runs the wrong way. Hadoop's original argument was that storage and compute on your own commodity hardware were far cheaper than a proprietary appliance. Object storage inverted that by letting storage and compute scale and be paid for separately, which is why new data platforms start on S3 or its equivalents rather than on HDFS. The commercial consequence is severe: Cloudera and Hortonworks merged, MapR was sold, and the free distributions CDH and HDP reached end of support, so the packaged, patched, freely available Hadoop that most organisations actually ran no longer exists. What remains is a subscription product from Cloudera or assembling Apache tarballs yourself. Who still runs it are organisations with an existing multi-petabyte on-premises estate, often for data residency, egress cost or regulatory reasons, where HDFS at rack scale is genuinely cheaper than cloud object storage. The trade-off is staffing and direction. Operating it is a specialism covering Kerberos, YARN queues, JVM tuning and a stack of interdependent project versions, the people who hold that skill have been moving to cloud data platforms for a decade, and MapReduce itself is maintained for compatibility rather than developed, so new work written against it is written against a dead end.

What people use it for

  • Operating an existing multi-petabyte on-premises estate where data residency or egress costs rule out moving to cloud object storage
  • Running Spark or Flink under YARN on hardware you already own, using HDFS as the storage layer
  • Keeping long-lived regulated archives on infrastructure entirely within your own data centres and legal jurisdiction
  • Maintaining legacy Hive and MapReduce workloads during a staged migration to a lakehouse or cloud platform

The honest half

Where it falls short

Concrete and checkable, so you can decide whether any of them matter to you. This is the half of a review a vendor will not write about Apache Hadoop.

  • The free vendor distributions no longer exist: Cloudera's CDH and Hortonworks' HDP have reached end of support and the successor CDP is subscription-only, so running Hadoop without paying now means assembling, testing and security-patching Apache releases yourself.
  • HDFS couples storage to compute, so adding capacity means buying whole nodes with CPU and memory you may not need, and the entire industry moved to object storage precisely because it lets the two be bought separately.
  • The NameNode holds all filesystem metadata in memory, so a cluster with tens of millions of small files exhausts heap long before it exhausts disk, and the remedy is a file compaction job that somebody has to write, schedule and own indefinitely.
  • Operating it is a distinct specialism covering Kerberos, YARN queue tuning, JVM garbage collection and the compatibility matrix between Hive, HBase, Ranger, Oozie and the core, and an upgrade touches all of them at once rather than one at a time.
  • MapReduce is maintained for compatibility rather than actively developed, and new work goes to Spark or Flink, so a job written against MapReduce today is written against an API that will not gain anything further.
  • Hiring is against you: the talent pool has moved to cloud data platforms over the past decade, so a Hadoop estate increasingly depends on a small number of individuals, which makes it a succession risk before it is a technical one.

Capabilities

Features

  • HDFS

    Distributed filesystem that splits files into blocks and replicates them across nodes, with erasure coding as a lower-overhead alternative

  • YARN

    Cluster resource manager that schedules containers across nodes, with queues, capacity guarantees and multi-tenancy

  • MapReduce

    The original batch processing model, retained for compatibility with existing jobs

  • HDFS federation and high availability

    Multiple namespaces and standby NameNodes with automatic failover to remove the single metadata point of failure

  • Kerberos security

    Cluster-wide authentication for users and services, with Ranger commonly layered on for authorisation and audit

  • Rack awareness

    Places replicas across racks so a rack failure does not take out every copy of a block

  • S3A and object store connectors

    Reads and writes cloud object storage through the Hadoop filesystem API, which is how many non-Hadoop tools inherited S3 support

  • Ecosystem compatibility

    Spark, Flink, Hive and HBase all run on YARN and read HDFS, so the cluster can host engines newer than MapReduce

Answered, with sources

Questions people ask

Each answer names the page it came from, so you can check it rather than take our word for it.

Is Hadoop dead?

No, but it is legacy. Large on-premises HDFS estates still run and are still supported, and Spark and Flink still run on YARN. What has ended is Hadoop as a default choice for new platforms, which now start on object storage.

Can I still get a free packaged distribution?

Not a maintained one. CDH and HDP reached end of support and Cloudera's CDP is a paid subscription. The remaining free route is building and patching Apache releases yourself, which is a real engineering commitment.

Do I need Hadoop to run Spark?

No. Spark runs standalone, on Kubernetes and on managed cloud services, and reads object storage directly. Many Spark deployments include Hadoop client libraries for the filesystem connectors without running a Hadoop cluster at all.

What replaced HDFS?

Object storage, typically S3 or a compatible system, combined with an open table format such as Apache Iceberg or Delta Lake. Apache Ozone exists as an object store within the Hadoop ecosystem for organisations staying on-premises.

Is it cheaper than the cloud?

It can be at multi-petabyte scale with steady, predictable utilisation, particularly where egress charges would be large. Include the staffing cost honestly, because the specialist operators a Hadoop cluster requires are scarce and therefore expensive.

Share

Keep looking

Where to go from Apache Hadoop

Best Databases software for

Compare Apache Hadoop with

Other Databases software

  • A distributed engine for batch, SQL, streaming and machine learning workloads over data that does not fit on one machine.

    Free plan11 researched notes
  • Long-established enterprise MPP data warehouse, rebranded in 2026 as the Autonomous Knowledge Platform, sold for cloud, on-premises and hybrid.

    Pricing on request10 researched notes
  • JMS message broker underpinning bank and telecom estates, now maintained rather than expanded

    Pricing on request12 researched notes
  • Streaming SQL engine built on ClickHouse internals, shipping as one small binary

    Free, then $199/mo12 researched notes
  • MIT-licensed analytical SQL database that runs inside your process, with no server, no dependencies and one writer at a time.

    Free plan10 researched notes
  • A distributed SQL engine that queries data where it already lives, across object storage, warehouses and operational databases.

    Free plan11 researched notes
  • Real-time data integration combining streaming, CDC, and batch

    Free, then $0.5/GB10 researched notes
  • Google Cloud's serverless analytical warehouse, billed either by bytes scanned per query or by reserved compute slots.

    Free plan13 researched notes
  • Document-relational database whose hosted service closed in 2025 and whose core is now unmaintained Apache 2.0 code.

    Free plan12 researched notes
  • Managed ClickHouse with a workflow that turns SQL queries into hosted HTTP APIs

    Free, then $25/mo12 researched notes
  • Centralised data access governance from the creators of Apache Ranger, now rebranding as Trust3 AI

    Pricing on request13 researched notes
  • Apache 2.0 platform for running many Postgres instances on Kubernetes, with copy-on-write branching and scale-to-zero.

    Free plan12 researched notes
  • Erlang MQTT broker whose source is Apache 2.0 but whose official binaries need a paid subscription

    Free plan12 researched notes
  • Distributed AI search platform for retrieval, ranking, and inference

    Free plan11 researched notes
  • Open source distributed SQL database for cloud native apps

    Free, then $0.1/hour13 researched notes
  • Managed vector database and vector lakebase for AI applications

    Free plan11 researched notes
  • Create apps that perfectly fit your team's needs

    Free, then $20/month per editor11 researched notes
  • The cloud-native distributed SQL database

    Free plan12 researched notes

Softwr does not host reviews and shows no star rating for Apache Hadoop, because a rating we did not collect is not ours to publish. What is here is the pricing and platform detail from the vendor’s own pages, limitations we could state concretely, and alternatives a reviewer confirmed people weigh against it. Tell us if any of it is wrong.

More on Apache Hadoop

Best Databases software alternatives

Industrial process historian with published per-tag pricing and no client licence fees

Per tag band per year

NetApp-owned managed service for Cassandra, Kafka, OpenSearch, PostgreSQL and Cadence with a bring-your-own-cloud model

quote

Erlang MQTT broker for large IoT fleets, relicensed to BSL with production free use limited to one node

Per month by connection and session volume

Attribute-based access control and masking applied inside Snowflake, Databricks and BigQuery

quote

MPP analytical database with a MySQL wire protocol and sub-second aggregation on wide tables

Open source, no licence fee

Streaming database that maintains incremental materialised views in SQL instead of Flink jobs

Per RisingWave Unit hour

Centralised data access governance from the creators of Apache Ranger, now rebranding as Trust3 AI

quote

The Meta-lineage distributed SQL query engine, distinct from the Trino fork

Open source, no licence fee

Compare Apache Hadoop with alternatives