How zomato manages it's logs?

Well zomato, is not only India's biggest palyer in food delivery market but it also south asia's biggest food delivery aggregator, serves millions of users daily, resulting in an enormous volume of log data. With a peak log rate of 150 million events per minute and over 50 terabytes of uncompressed logs generated each day, maintaining efficient and scalable logging infrastructure is obvious.

But how does they manage all this? Well, they recently made a big switch—from Elasticsearch to ClickHouse. I’ve been diving into how and why they did this, and it’s pretty fascinating.

#Challenges with Elasticsearch

Let’s start with Elasticsearch. Zomato initially used it for their logging needs, but... hmmm, things started to get tricky.

• Scaling Difficulties: Elasticsearch clusters were struggling to keep up with the rapid growth in log data. The constant need for over-provisioning to manage traffic spikes became a real hassle. Adding more nodes to the cluster wasn’t always smooth sailing—it often led to data rebalancing issues.

• Cost Implications: Managing and scaling Elasticsearch clusters turned out to be pretty costly. The need to handle variable traffic patterns meant that expenses were climbing, making the system less cost-effective overall. The heavy resources needed for indexing and querying large volumes of data didn’t help either.
• Performance Issues: As the data volume grew, Elasticsearch's performance started to lag. Slower query times and increased ingestion delays became a concern. With growing datasets, queries and aggregations took longer, affecting overall performance.

#Why ClickHouse?

So, engineering team at Zomato looked around for alternatives and eventually landed on ClickHouse. Why ClickHouse? Let’s break it down:

• High-Write Throughput: ClickHouse is designed for high-speed data ingestion. Its multi-threaded architecture is optimized for parallel processing, which makes it perfect for handling those massive log volumes.

• Column-Oriented Storage: ClickHouse uses column-oriented storage, which means faster read and write operations. This setup helps with efficient data storage and retrieval, especially useful for analytical workloads.

• Horizontal Scalability: ClickHouse’s shared-nothing architecture allows for easy scaling by adding more nodes. This setup reduces operational overhead and improves fault tolerance.

• Efficient Compression: ClickHouse employs advanced compression techniques like LZ4 and ZSTD. These techniques help minimize storage space and improve I/O efficiency, which is crucial when dealing with petabyte-scale data.

#The Logging System architecture

Zomato's logging system architecture

• Filebeat: This tool collects logs from Docker containers and EC2 instances. Filebeat forwards logs to Kafka, which helps in scaling and reliability.

filebeat.inputs:
  - type: log
    paths:
      - /var/log/*.log

output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092"]
  topic: "logs"

• Kafka: Acts as a buffer, ensuring reliable transmission of logs to ClickHouse. Kafka’s distributed storage and message queuing make it a solid choice for log data streaming.

• Custom Golang Workers: These workers process and batch logs from Kafka before they get into ClickHouse. They handle high throughput and transform log data for storage.

// Go code for processing and batching logs
package main

import (
    "github.com/segmentio/kafka-go"
    "github.com/ClickHouse/clickhouse-go"
)

func main() {
    kafkaReader := kafka.NewReader(kafka.ReaderConfig{ /* Kafka config */ })
    clickhouseConn, _ := clickhouse.OpenDirect("tcp://clickhouse:9000")

    for {
        msg, _ := kafkaReader.ReadMessage(context.Background())
        clickhouseConn.Exec("INSERT INTO logs VALUES (?, ?, ?)", msg.Time, string(msg.Value), "INFO")
    }
}

#Data Storage & Ingestion

So, the log data is now in ClickHouse, which is scaled to meet the needs. They initially set up with 10 M6g.16xlarge AWS EC2 nodes, though this could change as requirements evolve. These high-memory, high-compute EC2 instances ensure ClickHouse can handle large volumes of data efficiently.

ClickHouse buckets

• Batch Processing Ingestion: Instead of relying on ClickHouse Kafka plugins, custom Golang workers are used to batch log entries. Each batch contains up to 20,000 entries, which keeps ClickHouse efficient and reduces lag to less than 5 seconds.

• Native Format Ingestion: Data is inserted using ClickHouse’s native format, which boosts performance and reduces I/O intensity compared to HTTP-based insertion.

-- Create a table for logs
CREATE TABLE IF NOT EXISTS logs (
    timestamp DateTime,
    message String,
    level LowCardinality(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (timestamp);

-- Example of batch insertion using ClickHouse’s native format
INSERT INTO logs (timestamp, message, level) VALUES
('2024-07-29 12:00:00', 'User logged in', 'INFO'),
('2024-07-29 12:01:00', 'Error processing request', 'ERROR'),
('2024-07-29 12:02:00', 'User logged out', 'INFO'),
('2024-07-29 12:03:00', 'System update completed', 'INFO');

• Round-Robin Load Distribution: Workers use a round-robin method to distribute the load evenly across ClickHouse nodes. This helps in balancing the load and avoiding bottlenecks.

// Pseudo-code for round-robin distribution
nodes := []string{"node1", "node2", "node3"} // List of ClickHouse nodes
currentNode := 0

for _, logEntry := range logEntries {
    node := nodes[currentNode]
    currentNode = (currentNode + 1) % len(nodes)

    insertIntoNode(node, logEntry)
}

func insertIntoNode(node string, logEntry LogEntry) {
    // Function to insert logEntry into the specified node
}

#Schema Design in ClickHouse

Logs are stored in semi-structured tables. This design allows for flexibility in querying and efficient storage. The schema adapts to varying log formats and structures, which is pretty neat.

• Compression Optimization: The schema uses codecs and strings to enhance compression and reduce storage needs. Codecs like Delta, Gorilla, and LZ4 help achieve high compression rates.

-- Example schema with semi-structured tables and compression optimization
CREATE TABLE logs
(
    timestamp DateTime,
    message LowCardinality(String),
    level LowCardinality(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (timestamp);

• Custom SDK: Zomato built an SDK to enforce structured logging practices, standardizing top-level fields. This helps with consistent log formatting and improves query efficiency.

Secondary Indexes: They use the TokenBF_v1 Index, which improves query speed by bypassing non-matching data blocks. It’s useful for large-scale data scans.

-- Example of Bloom filter usage for optimizing query performance
CREATE TABLE logs
(
    timestamp DateTime,
    message String,
    level String
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (timestamp)
PRIMARY KEY (level)
INDEX idx_level (level) TYPE bloom_filter(0.1);

Query Throttling: They’ve implemented throttling mechanisms to handle high query loads and ensure system availability. This way, they can manage peak loads without sacrificing performance.

Metrics Collection: They opted for prometheus to scrape ClickHouse metrics and Grafana for visualization. This setup helps monitor system performance in real-time and set up alerts.

The move to ClickHouse has been a game changer for Zomato.

They’ve achieved real-time data ingestion with less than 5 seconds of lag, which supports swift decision-making. Plus, the system’s reliability has improved, making log data access almost instantaneous.

• Performance: Query times have dropped to a P99 of 10 seconds, and complex queries now finish in under 20 seconds. This enhancement is a big win for analyzing large data volumes.
• Cost Savings: Switching to ClickHouse has potentially saved over a million dollars annually compared to their old Elasticsearch setup. The cost-effectiveness of ClickHouse has definitely paid off.

It’s fascinating to see how they tackled the challenges of scaling and cost, all while improving performance. Now, they’re not just keeping up with their growing data—they’re staying ahead of it.

After lyft, uber and now after zomato's upgrade to clickhouse is an example of how adapting to new technologies can make a big difference and how clickhouse has that reliability and robust status quo and seal of usage by the top tech firms.