Softwr

Testing · head to head

Testsigma vs Umami

Testsigma logo

Testsigma

Testing

Natural language test automation for web, mobile and API with a self hosted open source edition

From
Free
Rated
-
Full reviewWebsite
Umami logo

Umami

Marketing

Open source, cookie-free web analytics you can self-host for the cost of a database

From
Free
Rated
-
Full reviewWebsite

The short version

  • Each has a real cost: Testsigma the natural language layer is a fixed vocabulary rather than real English, so testers spend the first weeks learning which phrasings the parser accepts, which is learning a language after all.; Umami there is no cohort analysis, retention curve, session replay or multi-touch attribution, so a marketing team that needs to justify channel spend will still buy a second tool and Umami becomes a supplementary dashboard.
  • They diverge on capability: Testsigma covers Natural language authoring, Umami covers Cookie-free tracking.
  • Prices and features above were last checked on 31 August 2026.

Where they differ

Only the attributes on which Testsigma and Umami actually diverge.

Attributes where Testsigma and Umami differ
AttributeTestsigmaUmami
Pricing modelPer user per monthPer month by event count
PlatformsWeb, iOS, Android, LinuxWeb, Linux
CategoryTestingMarketing

Identical on both: starting price (Free), free tier (Yes), user rating (Not yet rated).

What each one covers

Drawn from each product's published feature list. An absence here means we hold no record of it - not that the product lacks it.

Only in Testsigma

  • Natural language authoring
  • Unified web, mobile and API tests
  • Self hosted community edition
  • Cloud device and browser grid
  • Data driven runs
  • Custom functions

Only in Umami

  • Cookie-free tracking
  • Self-hosting
  • Small tracking script
  • Custom events
  • Funnels and goals
  • Multi-site and teams
  • Public dashboards
  • API access

What people use each for

The jobs each tool is most often brought in to do.

Testsigma

  • A team maintaining separate Selenium and Appium suites that wants one place for web and native mobile coveragenot Umami
  • Organisations that must keep test data and execution on premises for regulatory reasonsnot Umami
  • QA groups with manual testers being asked to automate without first learning Javanot Umami
  • Evaluating low code test automation properly before committing budget, by running the free edition against the real applicationnot Umami

Umami

  • A developer who wants traffic numbers on a personal or client site without adding a consent bannernot Testsigma
  • An organisation whose legal or procurement team has ruled out sending visitor data to a US advertising companynot Testsigma
  • A high-traffic site where hosted analytics priced by pageview band has become expensive and self-hosting removes the ceiling entirelynot Testsigma
  • A team that wants analytics data in a database they already control so they can query and join it themselvesnot Testsigma

Where each one falls short

Documented limitations, not opinions. Every one is a constraint you would hit in normal use.

Testsigma

  • The natural language layer is a fixed vocabulary rather than real English, so testers spend the first weeks learning which phrasings the parser accepts, which is learning a language after all.
  • Anything conditional or arithmetic falls through to custom Java or JavaScript, and that code lives in the platform editor rather than your repository, where it gets no code review and no version history you control.
  • The community edition lags the cloud product and is not the version the vendor demonstrates, so a self hosted evaluation can understate or overstate what you would actually buy.
  • Cloud execution is billed separately from authoring seats, so a suite that grows from two hundred to two thousand tests changes the bill without anyone changing the plan.
  • Support and engineering are concentrated in Indian time zones, which slows incident turnaround for teams working in the Americas.

Umami

  • There is no cohort analysis, retention curve, session replay or multi-touch attribution, so a marketing team that needs to justify channel spend will still buy a second tool and Umami becomes a supplementary dashboard.
  • Self-hosting is only free in licence terms; you own the database, the upgrades, the backups and the query performance at scale, and a busy site's events table grows quickly enough to need real attention.
  • Because it stores no persistent identifier, it cannot follow a user across devices or sessions in any meaningful way, which makes returning-visitor and lifetime-value questions unanswerable by design rather than by omission.
  • Cloud overage is billed per event beyond the plan allowance, so a traffic spike or a badly instrumented custom event can produce a bill materially above the $20 or $200 headline before anyone notices.
  • It is maintained by a small team, so the pace of new capability is modest and enterprise expectations such as SSO, granular role permissions and formal support agreements are limited compared with commercial analytics vendors.

Pricing, plan by plan

Testsigma

Free
  • Community Edition$undefined/month
    • Self hosted, open source, no licence fee
    • You supply the infrastructure and the browser or device grid
  • Cloud$undefined/month
    • Priced per authoring user with additional charges for parallel cloud executions and device minutes
    • Published tiers exist but the figures that matter at team scale come from sales

Umami

Free
  • Self-hostedFree
    • MIT licensed, no licence fee
    • No event limit imposed by the software
    • Unlimited websites and users
  • Cloud HobbyFree
    • Up to 100,000 events per month
    • 3 websites
    • 6 months data retention
  • Cloud Pro$20/month
    • Up to 1,000,000 events per month
    • Unlimited websites
    • Unlimited team members
  • Cloud Business$200/month
    • Up to 10,000,000 events per month
    • All Pro features
    • Higher volume overage rate

Which should you pick?

Choose Testsigma if

  • You need natural language authoring.
  • You want to start without paying.
  • You work on Web, iOS, Android, Linux.
  • You also want unified web, mobile and api tests.
Testsigma review

Choose Umami if

  • You need cookie-free tracking.
  • You want to start without paying.
  • You work on Web, Linux.
  • You also want self-hosting.
Umami review

Questions people ask

Is Testsigma or Umami better?
Neither clearly leads. Testsigma starts at Free and Umami at Free, and user ratings are close enough to be indistinguishable. Choose on capability and platform support.
Which is cheaper, Testsigma or Umami?
Testsigma starts at Free and Umami at Free.
Does Testsigma or Umami run on more platforms?
Testsigma runs on Web, iOS, Android, Linux. Umami runs on Web, Linux.
Can I use Testsigma for free?
Both have a free tier, so you can try either at no cost before committing.
What is Testsigma best used for?
Testsigma is most often used for a team maintaining separate selenium and appium suites that wants one place for web and native mobile coverage, organisations that must keep test data and execution on premises for regulatory reasons, qa groups with manual testers being asked to automate without first learning java, evaluating low code test automation properly before committing budget, by running the free edition against the real application. Of those, a team maintaining separate selenium and appium suites that wants one place for web and native mobile coverage and organisations that must keep test data and execution on premises for regulatory reasons are not what Umami is typically brought in for.
What can Testsigma do that Umami cannot?
Testsigma covers Natural language authoring, Unified web, mobile and API tests, Self hosted community edition, Cloud device and browser grid. Umami covers Cookie-free tracking, Self-hosting, Small tracking script, Custom events.

Answered from the vendors’ own pages

Testsigma: Is the open source edition genuinely usable in production?

Yes for self hosted execution, though it trails the cloud version on newer features and you maintain the grid yourself.

Umami: Is Umami really free?

Self-hosted, yes; it is MIT licensed with no event cap. You pay for the server and database. Umami Cloud has a free tier at 100,000 events a month and paid tiers from $20.

Testsigma: Do testers need to know Selenium?

Not to write ordinary steps. They will need someone who does as soon as the suite needs custom functions or awkward waits.

Umami: Do I need a cookie consent banner?

For Umami analytics specifically, generally no, because it sets no cookies and stores no personal data. Other tags on your site may still require one.

Testsigma: Can it test native iOS and Android applications?

Yes, through Appium underneath, on the hosted device grid or your own devices.

Umami: Can it replace Google Analytics?

For traffic, referrers and event counts, yes. For attribution modelling, audiences and advertising integration, no.

Umami: What does it need to run?

A container and a PostgreSQL or MySQL database. Most people deploy it on infrastructure they already pay for.

Share

Related pages

More on Testsigma

More on Umami

Keep looking

Other head to heads