How to reduce Datadog log costs without losing what you need

Log bills grow quietly. A service is scaled out, a debug flag is left on, a new health check is added, and the monthly invoice creeps up without anyone deciding it should. Datadog is a common place to notice, because its pricing makes every log line count twice. The ideas below apply to most log platforms, but the names are Datadog's.

Where the money goes

Datadog separates sending logs from making them searchable. Ingestion is charged per GB of logs sent. Indexing, which makes logs searchable for a chosen retention period, is charged per million log events, with longer retention costing more. Archives in your own cloud storage and Flex Logs offer cheaper ways to keep logs you rarely search. Check your contract for the exact rates; the structure is what matters here.

Two consequences follow. A log line you never needed costs you once to ingest, and again if it's indexed. And a single noisy source, such as health-check access logs, can be a large share of both.

1. Find the expensive patterns

Start by measuring. In Log Explorer, group by service and by status, then use the Patterns view to see which message shapes repeat most. The usual winners are health-check and readiness-probe requests, debug output, a line per successful request, retries in a loop, and whole objects dumped into one message. Datadog's estimated usage metrics (datadog.estimated_usage.logs.ingested_bytes and ingested_events) can be broken down by service to show where volume comes from.

2. Fix the source

The cheapest log is the one never written. Set production log levels to info or warning, turn off access logging for health endpoints in the web server or framework, and replace per-request success logs with metrics. This also saves CPU, network and agent work.

3. Filter before ingestion

Where you can't change the application, drop lines before they reach Datadog. The Datadog Agent supports log_processing_rules with exclude_at_match to discard lines matching a pattern at collection. Observability Pipelines, Vector, Fluent Bit or the OpenTelemetry Collector can filter, sample and reshape logs before they're sent. Lines dropped here are never ingested or indexed.

# Datadog Agent: conf.d/<integration>.d/conf.yaml
logs:
  - type: file
    path: /var/log/app/access.log
    service: web
    source: nginx
    log_processing_rules:
      - type: exclude_at_match
        name: drop_health_checks
        pattern: (healthz|readyz|livez|kube-probe)

4. Exclude from indexes

Index exclusion filters stop matching logs from being indexed, while they're still ingested, can still feed metrics generated from logs, and can still go to archives. Use them for logs you want to count or keep but rarely search. A filter can also sample: index 10 percent of successful requests and all errors.

5. Use more than one index

Not everything needs the same retention. Put audit and security logs in an index with long retention, and high-volume application logs in one with short retention. Retention is a large part of the indexing price.

6. Archive for compliance

Send everything to an archive in your own cloud storage (S3, Azure Storage or Google Cloud Storage) and rehydrate only what an investigation needs. This is how you keep long retention, such as the 180 days the CERT-In Directions require for ICT system logs in India, without paying to index it.

7. Turn logs into metrics

If you search logs mostly to count things (requests per endpoint, error rates, latency), generate metrics from logs at ingestion, or better, emit metrics from the application. Metrics are cheaper to store and faster to query, and you can then stop indexing the raw lines.

Guard against the bill growing back

• Set daily quotas on indexes so a runaway service can't blow the month's budget.
• Alert on estimated usage metrics by service.
• Review the top patterns monthly, and give each team its own numbers.
• Make log levels configurable at runtime, so debug can be switched on briefly and switched off again.

A worked example

A team sends 80 GB of logs a day. The Patterns view shows that load balancer health checks account for about 20 percent of events, a payments service left at debug level another 15 percent, and successful API requests 35 percent. Turning off health-check logging at the source and resetting the log level removes about a third of the volume before it's sent. An index exclusion filter that keeps all errors and 10 percent of successful requests cuts indexed events by about another third. Everything still goes to the S3 archive for 180 days. Ingestion falls by about a third and indexing by more than half, and investigators lose nothing they used.

Common mistakes

• Adding exclusion filters and expecting the ingestion bill to fall. Excluded logs are still ingested.
• Dropping logs that security or audit rules require you to keep. Archive them instead.
• Sampling errors along with successes, which hides the problems you most need to see.
• Cutting volume once and never looking again, while new services add noise every month.
• Indexing everything in one index with the longest retention any team asked for.

How to verify

Compare the estimated usage metrics by service for the week before and after each change. Check that alerts and dashboards built on logs still work, especially any that count events matched by an exclusion filter. Run a sample investigation, such as tracing one failed request end to end, to confirm the logs you kept are still enough.

Who should own log cost

Log volume is created by application teams but usually paid for by a platform or finance budget, so nobody who can change it sees the bill. The most effective long-term fix is organisational rather than technical: tag logs by service and team, show each team its own volume and estimated cost every month, and agree a budget or an index quota per team. When a team can see that its debug logging costs more than a new laptop each month, it tends to change the log level the same day. Make the cheapest path the default as well: a logging library configured with sensible levels, sampling for successful requests and structured fields, so new services start with good habits rather than inheriting noisy ones.

Checklist

• Top log patterns by volume identified and owned.
• Production log level set to info or higher.
• Health checks and probes not logged, or dropped before ingestion.
• Successful requests sampled; all errors kept.
• Index exclusion filters for low-value logs.
• Separate indexes with retention matched to need.
• Archive configured for long-term and compliance retention.
• Quotas and usage alerts in place.