Skip to contents

clean_table() makes the fixes that are unambiguously safe: it legalises column names and trims whitespace, and it reports near-duplicate labels without merging them. conform_table() is the next step and makes the two decisions that need judgement: deciding that "NSW" and "Nsw" are one category, and deciding that a character column holding six labels is a factor.

Usage

conform_table(
  df,
  merge_labels = c("report", "auto", "off"),
  types = c("report", "auto", "off"),
  max_levels = 20L,
  clean = c("auto", "report", "off"),
  quiet = FALSE
)

Arguments

df

A data frame.

merge_labels

One of "report" (default), "auto" or "off". Governs near-duplicate category labels, and treats the two kinds differently. A pair differing only by capitalisation is one category whatever happens, so it is merged onto the more frequent spelling, and on a tie onto whichever appeared first, which is deterministic and recorded. A pair differing by one edit might be two real categories, so it is merged only when one spelling is clearly the more common; on a tie it is refused and reported.

types

One of "report" (default), "auto" or "off". Governs storage format: a low-cardinality character column becomes a factor, a fully numeric character column becomes numeric, and a fully ISO-8601 character column becomes a Date.

max_levels

Cardinality at or below which a character column is treated as categorical. Default 20.

clean

Passed to clean_table(), which always runs first. Default "auto".

quiet

Logical. When FALSE (default) a cli summary is printed.

Value

An object of class masque_conformance: a list with

  • data - the conformed data.table;

  • cleaning - the masque_cleaning record from clean_table();

  • merges - one row per near-duplicate pair considered (col, from, to, n_from, n_to, applied, reason);

  • merge_map - the merges as applied to the data (col, from, to), one row per merged-away label, each to a label left in the column. This is what mask() records and apply_recipe() replays;

  • types - one row per column whose storage was considered (col, from, to, applied, reason);

  • assumptions - every decision as a plain sentence (col, assumption, applied), which is the thing to read;

  • modes - the modes applied;

  • max_levels - the cardinality threshold used.

Details

Both are off by default: the default "report" mode lists every change it would make, in plain words, and applies none. Set a mode to "auto" to apply them; the assumption behind each one is recorded in the returned object.

Storage is decided before categories, so a column of numbers or dates held as text is typed first and "4.2" is never merged into "4.4". Only columns that remain categorical are considered for merging, and a one-character edit between labels shorter than four characters is ignored ("a" and "b" are one edit apart).

Two labels that differ by an edit are merged only when one spelling is clearly more common; a column is converted only when every value parses; numeric columns keep their storage. Anything not changed for these reasons is reported as a not applied row with the reason.

Merges follow chains: if "Nsw" merges onto "nsw" and "nsw" onto "NSW", both end on "NSW". A label whose merges end on two different labels ("colur" beside "color" and "colour" that occur equally often) is not merged; it is reported, and a masque_conform_ambiguous_merge warning is raised under merge_labels = "auto". Codes that differ only inside a run of digits ("G01" and "G02") are never treated as near-duplicates. Which columns hold categories is decided the same way whatever types is, so types = "off" still merges labels and leaves storage as it is.

Pass the result to mask() (or as a table of mask_set()): the recipe records the merges and type changes, and apply_recipe() replays them on the original so a pipeline built on the conformed clone sees the same labels and storage.

See also

clean_table() for the safe fixes, propose_roles() for what a column is, mask() for the round-trip.

Examples

df <- data.frame(
  state = c("NSW", "Nsw", "NSW", "VIC", "VIC"),
  yield = c("3.1", "2.9", "5.0", "4.2", "3.8"),
  sown  = c("2024-05-01", "2024-05-03", "2024-05-01", "2024-05-08",
            "2024-05-08"),
  stringsAsFactors = FALSE
)
# report only: nothing is changed
cf <- conform_table(df, quiet = TRUE)
cf$assumptions
#>     col
#> 1 state
#> 2 state
#> 3 yield
#> 4  sown
#>                                                                                           assumption
#> 1 "Nsw" and "NSW" are the same category, recorded as "NSW" (differs from NSW by capitalisation only)
#> 2                                  stored as factor (was character): 3 distinct labels over 5 values
#> 3                                  stored as numeric (was character): every value parses as a number
#> 4                             stored as Date (was character): every value parses as an ISO-8601 date
#>   applied
#> 1   FALSE
#> 2   FALSE
#> 3   FALSE
#> 4   FALSE

# apply the decisions
cf2 <- conform_table(df, merge_labels = "auto", types = "auto", quiet = TRUE)
str(cf2$data)
#> Classes ‘data.table’ and 'data.frame':	5 obs. of  3 variables:
#>  $ state: Factor w/ 2 levels "NSW","VIC": 1 1 1 2 2
#>  $ yield: num  3.1 2.9 5 4.2 3.8
#>  $ sown : Date, format: "2024-05-01" "2024-05-03" ...
#>  - attr(*, ".internal.selfref")=<pointer: 0x55ef156a8a30>