Sampling & extrapolation

Scanning every object in a petabyte-scale bucket on every run would be slow and expensive, and it is not necessary. Argus scans a carefully chosen sample of each datastore and extrapolates to the full population using standard statistics - so you get an exposure estimate you can defend to an auditor without paying to read every file.

Two things make this trustworthy rather than hand-wavy: Argus scans the files most likely to matter first, and it records its estimates as a range with an explicit interval, not a bare number.

Choosing what to scan

Argus does not sample blindly. Before it reads anything, it ranks the objects in a datastore by how likely each is to contain sensitive data, using signals like:

• File type - structured data (CSV, JSON, spreadsheets, Parquet) and documents (PDF, DOCX) rank above source code or config.
• Name and path - names suggesting exports, backups, customers, or employees rank up; names suggesting tests, samples, or temp files rank down.
• Size and age - files in a realistic size range for real datasets, and recently modified files, get a nudge up.

A bucket’s name does not change how much of it is read or how its files are weighted: every datastore gets the same budget for its scan intensity.

Those weights are fixed rules, not a model that learns from your scans. The selection is a weighted draw rather than a strict top-N: a heavily weighted file is far more likely to be picked, but no file in the datastore has a zero chance, so the sample is biased toward finding real exposure without being blind to the rest.

Fair coverage

Because the draw is weighted rather than winner-takes-all, a large datastore’s low-ranked corners are still sampled from. Small datastores are scanned in full. Larger ones are sampled up to a per-datastore cap set by the scan intensity, so a very large datastore gets the same number of draws as a moderately large one; the drawer states what that sample can and cannot rule out. Each top-level folder and size range that holds a meaningful share of the objects or bytes is favoured so that, within the scan’s budget, each gets at least one draw where there is room. Gzip files are read; other compressed formats are counted as compressed and not read. A tenant-wide scan intensity setting (low, medium, or high) sets that cap, trading speed for thoroughness.

What was covered, and what it means for compliance

Every scan records what it actually read: readable objects sampled against readable objects present for an S3 bucket, with the unreadable ones counted by reason, and tables read against tables present for a database. The datastore drawer states it in one line (“Content sampled 25% (10 of 40 readable objects; 60 unreadable: 60 not a readable format)”) beside a High, Medium or Low confidence label.

A control’s status does not depend on how much of the datastore the sample covered. A control reads what its signals and the sample found, and its reason ends with one sentence on the sample it rests on (“Based on a 22% content sample (22 of 100 readable objects).”). A failure is never softened by coverage - a public bucket with sensitive data fails at any sample size. A scan that could read no content at all is the exception: its content controls read Not verified, never Compliant. A bucket holding only binary objects reads “Nothing to verify”; a pre-coverage scan states no figure.

Reading structured vs unstructured data

How Argus samples within a file depends on its shape:

• Structured data (CSV, JSON, XML) is sampled record-aware: Argus keeps the header and draws records from across the file - beginning, middle, and end - rather than just the first rows, so a sensitive column buried deep in the file is still caught.
• Unstructured data (PDF, DOCX, TXT) is sampled by pulling content from across the document rather than only the top.

Extrapolating to the whole

Once the sample is classified, Argus estimates what the full datastore contains using a Wilson score confidence interval with a finite-population correction - a well-established statistical method for estimating a proportion from a sample. Instead of a single guess, every estimate is recorded as a range: a point estimate plus a lower and upper bound, and a margin of error. The draw is weighted and capped by scan intensity, so read the range as an estimate, not a guarantee.

That is what lets Argus record “an estimated 12,000 records contain PII, within a stated range,” rather than a false-precision single figure. How a sample is judged too thin to trust is covered in Confidence scoring.

Incremental scans

Every scan is incremental, whether you start it, it runs on a schedule, or it follows discovery. A scan reads new and changed objects first, re-checks a sample of what was found sensitive before, and spends the rest of its budget on objects it has never read. Coverage builds up across scans: on a large bucket each scan reads new ground until the whole bucket has been read, and only then goes back over objects it already read. Argus remembers up to 5,000 objects per datastore this way; on a bucket with more, objects past that may be read again. A datastore’s first scan has no history, so it reads like a full sample, spreading its reads across the largest prefix and size groups. For databases, a table that has not changed since it was last read counts as read and is skipped.