
Bring a cleaned table to an analysable format, reporting every assumption
Source:R/conform_table.R
conform_table.Rdclean_table() makes the fixes that are unambiguously safe: it legalises
column names and trims whitespace, and it reports near-duplicate labels
without merging them. conform_table() is the next step and makes the two
decisions that need judgement: deciding that
"NSW" and "Nsw" are one category, and deciding that a character column
holding six labels is a factor.
Arguments
- df
A data frame.
- merge_labels
One of
"report"(default),"auto"or"off". Governs near-duplicate category labels, and treats the two kinds differently. A pair differing only by capitalisation is one category whatever happens, so it is merged onto the more frequent spelling, and on a tie onto whichever appeared first, which is deterministic and recorded. A pair differing by one edit might be two real categories, so it is merged only when one spelling is clearly the more common; on a tie it is refused and reported.- types
One of
"report"(default),"auto"or"off". Governs storage format: a low-cardinality character column becomes a factor, a fully numeric character column becomes numeric, and a fully ISO-8601 character column becomes aDate.- max_levels
Cardinality at or below which a character column is treated as categorical. Default
20.- clean
Passed to
clean_table(), which always runs first. Default"auto".- quiet
Logical. When
FALSE(default) aclisummary is printed.
Value
An object of class masque_conformance: a list with
data- the conformed data.table;cleaning- themasque_cleaningrecord fromclean_table();merges- one row per near-duplicate pair considered (col,from,to,n_from,n_to,applied,reason);merge_map- the merges as applied to the data (col,from,to), one row per merged-away label, eachtoa label left in the column. This is whatmask()records andapply_recipe()replays;types- one row per column whose storage was considered (col,from,to,applied,reason);assumptions- every decision as a plain sentence (col,assumption,applied), which is the thing to read;modes- the modes applied;max_levels- the cardinality threshold used.
Details
Both are off by default: the default "report" mode lists every change
it would make, in plain words, and applies none. Set a mode to "auto" to
apply them; the assumption behind each one is recorded in the returned
object.
Storage is decided before categories, so a column of numbers or dates
held as text is typed first and "4.2" is never merged into "4.4". Only
columns that remain categorical are considered for merging, and a
one-character edit between labels shorter than four characters is ignored
("a" and "b" are one edit apart).
Two labels that differ by an edit are merged only when one spelling is
clearly more common; a column is converted only when every value parses;
numeric columns keep their storage. Anything not changed for these reasons
is reported as a not applied row with the reason.
Merges follow chains: if "Nsw" merges onto "nsw" and "nsw" onto
"NSW", both end on "NSW". A label whose merges end on two different
labels ("colur" beside "color" and "colour" that occur equally
often) is not merged; it is reported, and a masque_conform_ambiguous_merge
warning is raised under merge_labels = "auto". Codes that differ only
inside a run of digits ("G01" and "G02") are never treated as
near-duplicates. Which columns hold categories is decided the same way
whatever types is, so types = "off" still merges labels and leaves
storage as it is.
Pass the result to mask() (or as a table of mask_set()): the recipe
records the merges and type changes, and apply_recipe() replays them on
the original so a pipeline built on the conformed clone sees the same
labels and storage.
See also
clean_table() for the safe fixes, propose_roles() for what a
column is, mask() for the round-trip.
Examples
df <- data.frame(
state = c("NSW", "Nsw", "NSW", "VIC", "VIC"),
yield = c("3.1", "2.9", "5.0", "4.2", "3.8"),
sown = c("2024-05-01", "2024-05-03", "2024-05-01", "2024-05-08",
"2024-05-08"),
stringsAsFactors = FALSE
)
# report only: nothing is changed
cf <- conform_table(df, quiet = TRUE)
cf$assumptions
#> col
#> 1 state
#> 2 state
#> 3 yield
#> 4 sown
#> assumption
#> 1 "Nsw" and "NSW" are the same category, recorded as "NSW" (differs from NSW by capitalisation only)
#> 2 stored as factor (was character): 3 distinct labels over 5 values
#> 3 stored as numeric (was character): every value parses as a number
#> 4 stored as Date (was character): every value parses as an ISO-8601 date
#> applied
#> 1 FALSE
#> 2 FALSE
#> 3 FALSE
#> 4 FALSE
# apply the decisions
cf2 <- conform_table(df, merge_labels = "auto", types = "auto", quiet = TRUE)
str(cf2$data)
#> Classes ‘data.table’ and 'data.frame': 5 obs. of 3 variables:
#> $ state: Factor w/ 2 levels "NSW","VIC": 1 1 1 2 2
#> $ yield: num 3.1 2.9 5 4.2 3.8
#> $ sown : Date, format: "2024-05-01" "2024-05-03" ...
#> - attr(*, ".internal.selfref")=<pointer: 0x55ef156a8a30>