DataSet (formerly known as Scalyr) is a high-performance, cloud-native data platform specializing in log management, observability, and real-time security data analytics. Acquired by cybersecurity leader SentinelOne in February 2021 for approximately $155 million, Scalyr was officially rebranded as DataSet in February 2022. Operating as a specialized business unit under SentinelOne, DataSet continues to serve enterprises globally from its headquarters in Mountain View, California.
The platform was originally founded by Steve Newman, the co-founder of Writely (which became Google Docs). Frustrated by the scale limitations, high costs, and slow query speeds of traditional index-based log management tools, Newman engineered a completely new database architecture from the ground up. Today, DataSet is designed to ingest, store, search, and analyze petabytes of machine data, application logs, and system events in real time, serving as a critical database solution for modern DevOps, SRE (Site Reliability Engineering), and security teams.
The Core Technology: A No-Index Database Architecture
Traditional log analysis and database solutions rely heavily on database indexing (such as Elasticsearch/Lucene-based platforms). While indexing speeds up specific queries, it introduces massive write-overhead, increases storage footprints, causes ingestion delays, and fails during high-cardinality search scenarios.
DataSet addresses these limitations through a proprietary, non-index, columnar database engine designed specifically for event data:
* Massively Parallel Query Engine: Instead of using indexes, DataSet uses a highly optimized, brute-force search algorithm that parallelizes queries across multi-tenant compute clusters. It can scan terabytes of data per second per query.
* Real-Time Ingestion: Because there are no indexes to build or update, data is available for querying within milliseconds of ingestion. This eliminates the "ingestion lag" common in traditional database architectures.
* High Compression and Columnar Storage: Data is stored in a highly compressed, columnar format, which reduces storage costs and optimizes memory bandwidth during analytical queries.
* Schema-on-Read: Rather than enforcing strict schemas at the time of ingestion, DataSet parses and structures data on-the-fly when queries are run, allowing for maximum flexibility as log formats change.
Products and Platform Capabilities
1. Enterprise Log Management
DataSet provides a centralized repository for all enterprise logs, including application logs, system events, cloud infrastructure audits, and network traffic.
* Multi-Cloud and Hybrid Ingestion: Supports seamless data ingestion from AWS, Google Cloud Platform (GCP), Microsoft Azure, on-premises servers, and containerized environments.
* DataSet Agent: A lightweight, open-source agent designed for minimal CPU and memory consumption. It aggregates logs, system metrics, and Kubernetes events, sending them securely to the DataSet SaaS platform.
* Open Integration Ecosystem: Integrates natively with standard collectors such as Fluentd, Fluent Bit, Logstash, Vector, and OpenTelemetry, ensuring companies do not face vendor lock-in.
2. Log Analytics and PowerQueries
DataSet features a powerful query interface designed for both rapid troubleshooting and complex data exploration.
* PowerQueries: A command-line-style query language that allows users to perform advanced data transformations, aggregations, joins, and calculations on-the-fly. Users can extract metrics from unstructured text, calculate percentiles, and group data dynamically.
* Natural Language and Faceted Search: Allows engineers to quickly filter logs using simple key-value pairs, free-text searches, or structured parameters without needing to learn complex syntax.
3. Real-Time Dashboards and Visualization
Visualization tools within DataSet convert raw log data into actionable business and operational intelligence.
* Dynamic Dashboards: Teams can build custom dashboards to monitor Key Performance Indicators (KPIs), system health, application latency, and error rates.
* Instant Refresh: Unlike traditional BI tools that require batch processing, DataSet dashboards refresh in real-time as new data points are ingested.
* Interoperability: Dashboards can be easily shared across teams, and users can drill down from high-level visual charts directly into the underlying raw logs with a single click.
4. Automated Alerting and Anomaly Detection
To prevent downtime and security incidents, DataSet includes a robust alerting engine.
* Low-Latency Alerting: Evaluates incoming data streams in real time, triggering alerts within seconds of an anomaly or threshold breach.
* Flexible Alert Definitions: Alerts can be based on simple thresholds, rate changes, absence of data, or complex PowerQuery expressions.
* Third-Party Notifications: Integrates with modern incident response and collaboration tools, including PagerDuty, Slack, Opsgenie, Webhooks, and VictorOps.
5. Kubernetes and Container Observability
Designed for microservices architectures, DataSet offers deep visibility into containerized workloads.
* Automated DaemonSet Deployment: Easily deploys across Kubernetes clusters to automatically collect stdout/stderr streams, container metadata, and node-level metrics.
* Context-Aware Troubleshooting: Automatically associates logs with specific pods, namespaces, deployments, and nodes, enabling developers to isolate container failures instantly.
Target Use Cases and Business Value
DevOps and Site Reliability Engineering (SRE)
For modern engineering teams, system downtime directly translates to lost revenue. DataSet serves as the primary debugging tool during active incidents. Its sub-second search times across terabytes of data allow SREs to isolate the root cause of application errors, microservice failures, or infrastructure bottlenecks faster than traditional index-bound tools.
Security Information and Event Management (SIEM) / SecOps
Under SentinelOne, DataSet's core technology has been integrated into SentinelOne's Singularity XDR (Extended Detection and Response) platform. However, as a standalone database solution, DataSet allows security teams to ingest and retain massive volumes of security telemetry (such as firewall logs, DNS requests, and active directory events) at a fraction of the cost of traditional SIEMs. This ensures security analysts have historical access to long-term data for threat hunting and forensic investigations.
Cost and Scale Optimization
Many companies scale back their logging practices—often discarding valuable debugging data—due to the prohibitive licensing and infrastructure costs of legacy database platforms. DataSet's architecture significantly lowers the total cost of ownership (TCO) by reducing hardware overhead, simplifying data pipeline administration, and offering flexible pricing models based on daily ingestion volumes without indexing penalties.
