*! version 3.2.1 13Aug2013
help validly

Title

validly -- generate or replace a variable, or run conditional commands,
correctly handling missing values in logical/relational expressions

Corrects Stata's mishandling of logical/relational missing values

Why this is necessary is explained in the Description and, more fully, in the
Discussion section below.

Index

(clickable links, use back-arrow button on Viewer to return)
Syntax
Description
Options

†† Discussion - [†† explains why it matters]

Examples
Method
Functions

Utilities
Restrictions
Error messages
Saved results
Reference
Contact

Syntax

As generate or replace, but, in contrast to unmodified Stata,
using valid functional forms for any logical or relational expression:

validly generate [type]
newvar [:lblname]
= exp
[if] [in]
[,options]

validly replace
varname
= exp
[if] [in]
[,options]

There is also an extension of substantive syntax,
to handle "condition False" and "condition indeterminate",
in addition to the standard "condition True" states; restricted, if desired, to a subset of observations:

validly generate|replace
var
= exp
[if] [in]
[, ifnot(expN) else(expE) when(expW) options]

Simplifications: validly can be abbreviated to vy;
replace can be abbreviated to rep;
and generate/replace can be completely omitted and are then imputed from
the existential status of var.

Also as assert, but using valid functional forms for any logical or relational expression:

validly assert
exp
[if] [in]
[,options]

For other non-assignment conditional commands, validly can act as a modifier or 'wrapper':

validly
conditional_command [, its options]
[,, validly's_options]

validly locates the conditional expression, replaces it
by a valid functional form, and executes the `wrapped' command (see example).

(Slightly inelegantly, but of necessity, validly's
options appear after double commas, to differentiate them from the command's options)

The functional macros, used implicitly in the valid commands,
can also be explicitly requested, in one of two forms:

validly global
mname
exp
[, options]

validly global
mname if exp
[, options]

Both return global macro $mname. The first form returns a version of exp evaluating validly to T, F or missing,
The second form treats missing values (from a validly evaluated exp) as False, and returns "if exp", so defined, to constrain Stata
to select only when the condition is in fact true. (See
below for examples of first form and second form use.)

Main
detail displays intermediate and variously helpful information;

else(expE) returns expE if not otherwise determined by condition;

extended preserves extended-missing values in logical/relational operations;

ifnot(expN) "if condition is not true" return expN;

nopromote see replace, options

source identifies the variable sources for missing values

when(expW) restricts to those cases where expW (validly interpreted) is True;

widely treats indeterminate (validly evaluated) conditionals as true.

assert
detail displays intermediate and variously helpful information;

fast see assert;

null see assert;

possible asserting that expressions which might-possibly-be-true (i.e. indeterminate) are acceptable;

rc0 see assert;

widely treats indeterminate (validly evaluated) conditionals as true.

global
detail displays intermediate and variously helpful information;

extended preserve extended-missing values in logical/relational operations;

widely treats indeterminate (validly evaluated) conditionals as true.

by is allowed with validly generate/gen/replace/assert, and when using validly as a 'wrapper' (but not with global); see [D] by.

+

| goto Index
+

Description

Think of "validly" as a modifier which corrects errors in certain commands.

Stata handles missing values intelligently within algebraic operations (so the sum of, say, 4 and an unknown number, is correctly deemed to be unknown). This intelligence does not
extend to logical and relational operations. The conjunction of two unknown values is, for example, deemed
by Stata to be a known true statement. There are workarounds (which have to be more complex
than they are usually presented as being) but they make life unnecessarily difficult when analysing real data.

To remedy this, validly generate has the functionality of generate, but,
in contrast to Stata's generate, behaves appropriately
when missing values are encountered within logical or relational expressions.

Likewise validly replace …

Likewise validly assert …

Likewise validly global …

Likewise validly reg …… if … (works for any other conditional command).

Consider a very simple example:

.generate pvq = p|q

this reports 'p or q' as True when both p and q are unknown. It should instead, obviously, report 'unknown'.
A simple "test for missing" does not resolve the issue:

.generate pvq = p|q if !mi(p,q)

will report 'p or q' as 'unknown' when p is True and q is unknown; it should report True (given a True-p, the disjunction is
known-True whatever the value of q). The invocation of validly allows the user to get the correct answer,
without having to craft more elaborate tests for missing, and without using multiple commands:

.validly generate pVq = p|q

The case for validly becomes more persuasive as the logical/relational expressions become more complex. There is a fuller exploration below in the Discussion section.

Note: Since validly handles appropriate "tests for missing values"
(within logical and relational expressions)
it is important not to combine the use of
validly with additional conditional tests for missing-values; the expressions should by typed as they are intended to be understood.

validly conditional_command executes the specified conditional_command,
but, in contrast to Stata's execution of the 'unwrapped' command, gives the correct result
when missing values are encountered within relational or logical expressions in the condition.

The generated valid functions, implicitly accessed by the above commands, can be generated and stored in global macros by validly global.
An example below gives an illustration where this might be useful.

Though the primary focus of validly is on correcting Stata's treatment of logical and relational expressions, as a by-product it allows extension of the substantive generate/recode syntax. Thus:

.validly gen v = a if b, ifnot(c) else(d)

will set variable v to 'a' when the condition 'b' is known-True; to 'c' when the condition is known-False;
to 'd' when condition 'b' is indeterminate. See an example of its use, below.
There is also a further syntactical extension:

.validly gen v = a if b, ifnot(c) else(d) when(e)

which evaluates across observations for which condition 'e' is known-True.

+

| goto Index
+

Options

Main

detail is a comforting reporting option (standard
report is on a purely 'need to know' basis),
which displays detailed information on the decisions being taken by validly -- specifically (see Method below) it
reports the RPN versions of
the expressions and the nested cond functions into which they are compiled (for handling missing values correctly).

[Inspection of the RPN code can help reassure that validly is doing what you intend;
even if detail is not set, this code is returned in r(RPN),
and can be inspected by the simple command vy (with no arguments.]

detail also increases informative reporting with other options (such as extended or widely).

Detailed reporting can be set as the default by set_detail.

else(expE) after an if, can be read as an option, or as an extension of the substantive recode syntax.
As its name suggests, it assigns expE (validly interpreted) to the result variable when the condition is not otherwise specified.
Often used in conjunction with ifnot. Thus:

.validly gen v = a if b, ifnot(c) else(d)

will set variable v to 'a' when the condition 'b' is known-True; to 'c' when the condition is known-False;
to 'd' when condition 'b' is indeterminate.
When option ifnot is omitted, else mops up all observations where condition is not known-True. Thus:

.validly gen v = a if b, else(d)

will set variable v to 'a' when the condition 'b' is known-True, and to 'd' otherwise (where the condition is false, or indeterminate).
Can be used with option when; also with option
extended to track extended-missing values through the expressions; it is obviously NOT compatible with widely.

extended retains as much information as possible about extended-missing values in relational/logical operations.
(By default validly treats extended-missing values as simply missing, sysmiss;
this is what plain Stata does in algebraic expressions, see Note 2.)

Consider a logical/relational operator which, on an observation, is due to return a missing value, since one operand is valid but the other carries a specific extended-missing value.
If preserving information, we would then expect the extended-missing value (the "reason" for the indeterminacy) to be carried forward to the result.
(Indeed, should both operands be indeterminate, but with identical extended-missing value, that value should be carried through to the result.)
This extended does.

Note 1: extended is an option, not the default (but see set_extended, below), since it is computationally expensive.
With extended set, the necessary cond
interpretation of expressions becomes markedly more complex, so the resulting code
is slower to execute -- see the discussion of functions -- and in rare cases may exceed Stata's capacity.

Note 2: Whilst Stata is good at preserving missing values in arithmetic operations, it is not good at preserving the detail of extended-missing values.
These, as the Stata help file says, "can be used to specify different reasons that a value is unknown". But consider:

.a + 4

where clearly that sum
should be regarded as unknown for the "reason" coded by '.a'; but Stata smoothes the result to sysmiss, willfully discarding information.
Thus the effect of the extended option (which does, for example, see '.a & T' as yielding '.a') may be vitiated by Stata's "smoothing" to sysmiss within any arithmetic operations involved.
A reminder of this is given if relevant operators occur.
Though functions (such as cond) can preserve extended-missing values, mostly they do not, so they also attract a reminder.

We note below how -- expensively -- extended-missing values can be helped to survive arithmetic.

Note 3 Strings. If the operands for a relation are of type string, the algorithm given above for extended makes no sense (string variables have only one possible
missing value, the null string, and cannot return that value to a numeric variable).
So validly checks for such operands, and for these substitutes an alternate encoding of the relation, which returns the extended-missing value .s
when the string-relation is indeterminate
(users can override that choice).

Note 4 extended always applies to the assigned expressions (the one immediately after the '=' sign, and the expressions within ifnot and else);

but the extended interpretation of an inclusion condition (the expression following an "if")
would be substantively well-defined only for generate
(without option widely). Consider,
as an example of where it is well defined:

.validly generate v = exp if expC, e

for an observation for which expC evaluates to a specific extended-missing value (say .x). Then variable v for that observation is assigned value
.x
if exp, for that observation, is determinate (or would itself also return value .x)
-- it would then be appropriately seen as indeterminate for "reason" .x (otherwise it,
as usual, remains plain sysmiss.

The implementation is discussed below.
Within replace (or for generate with option widely),
extended-missing values in 'expC' are smoothed to sysmiss.

Note 5 In default mode (without option extended) validly
smoothes extended-missing values around all operators (arithmetic, relational, logical) to sysmiss;
but if simple variables are assigned (as in '.vy v =p if r&s, ifn(q)') their extended-missing values are (where appropriate) passed-through to the result.

ifnot(expN) after an if, can be read as an option, or as an extension of the substantive recode syntax.
As its name suggests, it assigns expN (validly interpreted) to the result variable when the condition is Not-true. Thus:

.validly gen v = a if b, ifnot(c)

will set variable v to 'a' when the condition 'b' is known-True, to 'c' when the condition is known-False;
for other observations the variable is left 'as is' (here sysmiss. Since 'a' or 'c' may return missing, variable v may also contain other missing values.
Can be used with option else; also with option when;
and extended may be specified to track extended-missing values; it is compatible with widely.
See also the example below of its sequential use.

nopromote -- see replace, options (this is an option retained from standard Stata).

source aims to identify the sources (in terms of variables) of missing values in the result.
Each numeric variable, having missing values, appearing as the operand (in a logical or relational expression) is treated as if
all its missing-values were some variable-specific extended-missing value.
(These values, from .a upwards, are assigned in the order in which the variables are first encountered.)
Then, following the logic of extended, these notional values are propagated through the logical and relational expressions.
If a variable appears within an algebraic expression, this information, as discussed above, is lost;
further, source does not track individual string variables.
Selecting detail will yield summary table.

source(r) as source, but now conflicts of source are randomly resolved. Thus, for an observation for which we have '.a & .b' , source(r) randomly returns .a or .b
(with equal probability); plain source would
there return sysmiss. See below for example and discussion.

source(.x) and source(.x,r) are variant forms that allow choice of the starting value .x
for the source-code assignment,
which otherwise defaults to .a;
thus source(.a) is equivalent to plain source,
whilst source(.a,r) is equivalent to source(r);
for use of the variant form see example below.

Note 1 source may be most intelligible when issued with generate (or a 'complete' replace i.e.
one that is unconditional, or conditional with option else);
in other modes the result variable can obviously contain 'original' as well as 'source' missing-value codes.

Note 2 source, logically, affects the interpretation of any explicit extended-missing references

when(expW) can be read as an option, or as an extension of the substantive recode syntax.
It restricts the application of generate/ replace to those observations when expW (validly interpreted)
is True. It is
intended for use in conjunction with ifnot and/or else, where we might wish to apply these to a subset of the data.
Obviously, logically:

.validly v = a if p, when(expW)

.validly v = a, when(p & expW)

.validly v = a if p & expW

would all be identical in effect.
In contrast:

.validly v = a if p & expW, ifnot(b) else(c)

.validly v = a if p, ifnot(b) else(c) when(expW)

are not identical in effect. Both assign 'a' when 'p & expW' (validly interpreted) is True.
But they diverge on the assignment of 'b'. The first assigns 'b' when 'p & expW' (validly interpreted) is false;
the second assigns 'b' only to observations for which p is false whilst expW is true.
Also on 'c'. The first assigns 'c' when 'p & expW'
(validly interpreted) is indeterminate; the second assigns 'c' only to observations for which p is indeterminate
whilst expW
is true.

Note 1 In the presence of else, replace with option when(expw)
redefines all - and only - those observations for which expW is true.
See below for example and further discussion.

Note 2 when always selects-for-True, and so is unaffected by option widely. Thus:

.validly v = a if p, when(expW) w

.validly v = a if p & expW, w

are not be identical in effect
(the first assigns 'a'when 'p' is True-or-indeterminate and 'expW' is True; the second assigns when 'p & expW'
is True-or-indeterminate).

widely sets a 'wide' inclusion policy when applying a conditional -- the 'if' expression is validly
evaluated, but then (in contrast to validly's standard practice),
both true and indeterminate observations are selected.

Note: As with possible, though "include if missing" looks equivalent to the unvarnished Stata rule, the overall effect is not equivalent to unvarnished Stata; that is:

.gen y = 4 if p|!q

is NOT the same as:

.validly gen y = 4 if p|!q, widely

because validly first evaluates p|!q correctly.

Assert

detail -- as above

fast -- see assert

null -- see assert

possible treats observations where the asserted expression is possible (that is, it evaluates to indeterminate) as not
contradicting the assertion. Think of the command as saying:
"The assertion is true or possible".
Contrast this to validly's standard mode, where it is considered unacceptable to assert something as true if its value is
sometimes indeterminate. This is discussed further using an example, below.

Note: As with widely, though possible looks equivalent to the unvarnished Stata rule, the overall effect is not equivalent to unvarnished Stata; that is:

.assert p|!q

is NOT the same as:

.validly assert p|!q, possible

because validly first evaluates p|!q correctly.

rc0 -- see assert

widely -- as above

Global

detail -- as above

extended -- as above, but not for "validly global mname if exp"

widely -- as above, but only for "validly global mname if exp"

+

| goto Index
+

Discussion

As noted above, Stata handles missing values intelligently within algebraic operations (so, for Stata, the sum of, say, 4 and an unknown
number, is correctly deemed to be unknown). This intelligence does not
extend to logical and relational operations.
In logical and relational expressions, Stata treats missing values not as missing but as being very-large-numbers
-- see missing and relational operators and if exp.
It also treats all non-zero values as 'true'. That second rule is, by itself, innocuous.
But the combination of these two rules generates some bizarre consequences.

The table below shows how Stata's raw generate interprets logical operators: !p (variable np),
p&q (variable paq), p|q (variable pvq); and an exemplar equality relation p>=q (variable pgeq) -- for the
nine possible combinations of values of indicator variables p, q.

For example: for Stata, if both p and q are unknown (row nine), 'p and q' is nevertheless deemed to be true.
Asserting combined unknowns to be true is the stuff of fantasy novels, not data analysis.
The other red values in the table list other Stata errors.
(Notice, for example, that for Stata, where p is an indicator variable,
!!p is not the same as p.)

In contrast, the variables Np, pAq, pVq and pGEq have been constructed using validly
generate. As can be seen, they preserve the correct formal (and intuitive) behaviour
of logical and relational operators in the presence of missing values.

!p p&q p|q p>=q
+--------+-----------------------------------------------+
| p q | np Np paq pAq pvq pVq pgeq pGEq |
+--------+-----------------------------------------------+
1. | 1 1 | 0 0 1 1 1 1 1 1 |
2. | 1 0 | 0 0 0 0 1 1 1 1 |
3. | 1 . | 0 0 1 . 1 1 0 . |
+--------+-----------------------------------------------+
4. | 0 1 | 1 1 0 0 1 1 0 0 |
5. | 0 0 | 1 1 0 0 0 0 1 1 |
6. | 0 . | 1 1 0 0 1 . 0 . |
+--------+-----------------------------------------------+
7. | . 1 | 0 . 1 . 1 1 1 . |
8. | . 0 | 0 . 0 0 1 . 1 . |
9. | . . | 0 . 1 . 1 . 1 . |
+--------+-----------------------------------------------+
[Variables with uppercase code valid operators,
lowercase are plain Stata]

Stata's plain commands can of course be persuaded to "check for missing values", but the simple rule scales-up much less intuitively
than standard presentations might indicate.

For example: the Stata intention "generate v = p|q" cannot be handled simply by "adding a check for missing values". The command:

.generate pvq = p|q if !mi(p,q)

does not work. For example, it assesses '(T|.)' as sysmiss, when the correct value is T.
In unvarnished Stata, we have to check for missing values
within each arm of the disjunction.
Then, within unvarnished Stata, we have a choice of strategies.
We could use
separate commands for false and true, thus:

.generate pVq = 0 if !(p|q)

.replace pVq = 1 if (p&!mi(p)) | (q&!mi(q))

Or we could wrap it all into one expression:

.generate pVq = p|q if !mi(p,q) | (p&!mi(p)) | (q&!mi(q))

These are the simplest workarounds available in standard Stata.
Neither is intuitive to write; neither is remotely easy to read.
In contrast validly just "writes it as it is".
This makes it much easier to understand, and so to check, what is being intended. That matters
when you come to review your own analyses;
your future self also will be grateful.
We can give the clear, one-line, command:

.validly generate pVq = p|q

and the point holds more strongly for more complex initial expressions.

Logical operators within validly are interpreted as in the valid (black, uppercase) columns of the table.
Both logical and relational operators return 1 for True and 0 for False. Thus:

!p is interpreted as in Np

p&q is interpreted as in pAq

p|q is interpreted as in pVq

Relational operators within validly all share the missing values pattern of pGEq

[with the one
exception that when the relational operand explicitly references a specific extended-missing code (e.g.
'p > .b') the relation evaluates only to T or F.
Though here the equality relation is treated slightly differently -- see the example discussed below.
Also heed the caution below on the use of such explicit forms.]

For relational operators it might seem that Stata's conventions (treating missing values as alphabetically ordered large numbers) are a lesser problem -- so, if considering p>q
the mid-range user might suppose we just have to add: if !mi(p,q) ??
But even here the room for error is subtle and easily misread. For example:

.validly replace y = p>q if r==0

would do exactly what it says
on the tin ("For observations for which r is zero, replace y by the value of p>q -- whether that be T, F or indeterminate"). In contrast, the single
unvarnished Stata line:

.replace y = p>q if r==0 & !mi(p,q)

would not suffice; we would have had to have preceded that by the additional command:

.replace y = . if r==0 & mi(p,q)

Not all users would have done this. Which, again, is why Stata's default conventions are dangerous.

Extensions: Since, to perform its corrections on unvarnished Stata, validly has to disaggregate the expressions within generate/replace,
it can extend the "functionality" (horrid word) of these commands.
The extensions are implemented using standard Stata conventions for a command's options; you can view the device as "options", or as extensions of the generate/replace syntax. Thus:

.validly [generate|replace] v = a if b, ifnot(c) else(d)

will set variable v to 'a' when the condition 'b' is known-True; to 'c' when the condition is known-False;
to 'd' when condition 'b' is indeterminate.

There is also a further 'syntactical' extension:

.validly [generate|replace] v = a if b, ifnot(c) else(d) when(e)

That evaluates the generate/replace across observations for which condition 'e' is known-True.

See the example below, using all three options.

A further extension is that validly can track the data's extended-missing values through logical/relational expressions, or
identify the variable sources of indeterminacy.

For an explication of the mechanics behind validly's simplification see the sections on method and on functions;
for deployment see examples of use in the next section;
and for a more extended discussion of the underlying issues, refer to the article cited below.

Note that related, less serious, issues affect functions min and max -- see example below.

Reminder: Since validly handles correctly "tests for missing values" it is important not to combine the use of
validly with additional conditional checks-for-missing
aiming at the same goal
-- such as calls to mi() or
other explicit missing-values tests within the conditional -- referencing the operands of the logical and relational expressions.
Leave validly to do that work. Thus:

.validly gen y = p&q if !mi(p)

would yield a nonsense (seeing '.|T' as sysmiss rather than T). But deployed judiciously, and with care,
explicit conditional tests for missing can be useful
(which is why, on encounter, validly issues a warning but not an error
message). For example:

.validly gen y = p&q if !mi(r)

would be fine, giving us valid p&q for observations on which r was defined.
A more extended instance of an appropriate deployment of
a conditional test for missing appears in the example below

+

| goto Index
+

Examples

|a| Straightforward |h| use of source

|b| impute command |i| on source(.x)

|c| explicit extended-missing |j| global

|d| conditional commands |k| assert

|e| repeated condition |l| global, syntax

|f| sequences (ifn, else, when) |m| [type] and value-label

|g| more on ifn, else, when |n| on max and min

You can think of validly (or, shortened, vy) as a command modifier which corrects errors in certain commands.

+-+

|a| Examples of validly generate or validly replace, used where generate or replace would (incorrectly if the relational/logical operators encounter missing values)
normally be found:

.validly gen pAq = p&q, detail

.validly generate v = !((p&q)|(p|q))&(r|!s)&t

.vy replace v = sqrt(inc) if !( class!=3 | class!=6 ), d

.vy gen lagnp = !p[_n-1]

+-+

|b| As a minor typing reduction, validly can impute whether generate or replace is intended. Consider:

.vy fresh = (assets/(incm - expend)>100) & (AgeF==AgeM), d

If variable fresh does not exist this is equivalent to validly generate; if it does exist, this is equivalent to validly replace.
If 'replace' is imputed this is clearly reported
(should you want the convenience of imputing generate without the perceived risk of imputing replace
there is a utility to so restrict).

+-+

|c| Explicit extended-missing references. Where these appear, as in:

.vy defaulter1 = debt>.b & male

validly's standard approach to a relation ("it is indeterminate if either operand is indeterminate") is obviously inappropriate.
Such relations are therefore treated literally
(it is assumed that would be the user's intention);
thus 'debt>.b' is true for those whose debt is unknown for reasons .c,,d,… and false otherwise.

For equality ('==') the reasoning is
slightly different. Consider:

.validly gen defaulter2 = (debt==.a|debt==.b) & age>55

The thinking here is as follows: The expression 'debt==.a' can be read as saying "debt is unknown for
reason .a". So obviously False if debt is known, or unknown for reason .b. But, on this reading, a coding
of 'sysmiss' for debt would reflect "an unknown unknown"
-- debt is unknown, perhaps for reason .a, perhaps for
reason .b. So it is proper, for an observation
with the value of sysmiss, to regard the assertion "debt is unknown for reason
.a" as indeterminate. So, for this explicit equality test against extended-missing values, validly returns T
or F unless sysmiss is encountered (when sysmiss is returned).

Implementation is described here. Should you nevertheless wish the bare explicit function (returning only T or F)
use plain Stata to generate the corresponding variable (.gen debta = debt==.a) and deploy that within validly.

Note that if option source is present,
explicit extended-missing references may not deliver as intended.

+-+

|d| Suppose we are concerned to run the following regression
(where incm may equal expend, and any variable may contain missing values):

.reg assets incm if (assets/(incm-expend)>100) & (AgeF==AgeM), b

That Stata expression will not extract the expected subset (for example, it may include observations where both
male and female ages are unknown). validly's ability to "wrap" a Stata command allows
us to carry out the above regression with one single command:

.vy reg assets incm if (assets/(incm - expend)>100) & (AgeF==AgeM), b

The same ploy can helpfully get other commands correct; for example:

.validly list a b c if p|!q

Four footnotes:

1 The beta option for reg, above, appears after one comma; were we to deploy validly's options (such as detail) these would appear after two commas (a pair not-separated by spaces),
though if habitually you invoke detail it might be simpler to invoke .vy set_detail once.

2 The valid condition from the last used 'wrapped' command is always returned in the macro r(ifcond),
so, for example, after the above command, we could display further similarly selected lists,
or run a regression on the same subset,
without re-invoking validly, thus:

.global cond `r(ifcond)'

.list d e $cond

.reg a b c $cond

3 'wrapped' commands return their own r() and e() saved
results exactly as when issued normally. Should the command be an estimation command, then the command-line used by validly
can be inspected by:

.di "`e(cmdline)'"

4 For 'wrapped' commands, any 'by' option is written preceding validly, thus:

.by country: validly reg … if …

+-+

|e| Where a clutch of commands, all qualified by the same conditional, is being run, it may well be convenient to
define (and then repeatedly re-use) a
valid macro containing the conditional. Reverting to the regression example above, we could, for example, do:

.validly global subset if (assets/(incm - expend)>100) & (AgeF==AgeM)

.reg assets incm $subset, b

.reg assets incm age $subset, b

.reg ……

This strategy avoids risk of retyping errors;
and gives the one 'point of change' (that definition of global $subset)
should you subsequently decide to change your inclusion criterion, and rerun your analyses.

Note: since, as detail will reveal,
the condition macro, in the absence of widely,
is coded in "missing->False" form, it is not suitable for re-use within validly.

+-+

|f| Handling sequential conditionals: [This is an example of the utility of validly's extensions to
standard generate/ replace syntax.]
Suppose we wished to capture an intention which might be informally expressed:

" if test1-OK, assign1; elseif test2-OK, assign2; else assign3 "

In the presence of missing-values this would not be well captured by use of the standard generate/ replace syntax:

.validly generate v = .z

.validly replace v = assign1 if test1

.validly replace v = assign2 if test2 & v==.z

.validly replace v = assign3 if v==.z

because observations for which test1 computed as indeterminate could be assigned values depending upon test2 or test3;
not our intention. Reverting to unvarnished generate/replace will obviously not solve this problem. But the following
sequence, using the validly extensions, does capture our intention:

.validly generate vt = assign1 if test1, ifnot(.z)

.validly replace vt = assign2 if test2 & (vt==.z), ifnot(vt) else(.)

.validly replace vt = assign3 if vt==z.

This correctly runs test2 on those known to fail test1, and returns assign3 only for those known to fail both.
Those who score indeterminate on the first test remain at sysmiss; those who, when tested, score indeterminate on test2 are set to sysmiss.
This pattern of missing values is an optimum solution, declining to pronounce on those for whom we cannot pronounce.
The solution 'scales-up' straightforwardly to longer 'elseif' chains, by reproducing the structure of the second line.

Suppose however we wished to use option extended,
to trace extended-missing values, the above solution no longer works
(the else in the second line smoothes to sysmiss, and replacing its argument by test2 does not do what is required).
We should now do:

.vy gen ve = assign1 if test1, ifn(.z) e

.vy rep ve = assign2 if test2, ifn(.z) else(test2) when(ve==.z) e

.vy rep ve = assign3 if v==z. , e

The substantive results are as before, but now extended-missing values, interpreted as "reasons for indeterminacy" are,
as far as possible, retained.
As before the solution 'scales-up' straightforwardly to longer 'elseif' chains, simply by reproducing the structure of the second line.

Four footnotes:

1 If extended is enabled, choose, for ifnot(.z), an extended-missing value not naturally present in the data.

2 Since validly has some of the characteristics of a concerned, but not always percipient, minder it frets
about tests-for-missing within conditionals (since, as noted above, they can generate nonsense)
so the first solution will result in a 'warning', which can be disregarded.

3 The else in the second line of the first solution
retains restricted scope, without encountering the problems when aims to address,
because, in that particular instance, 'test2 & v==.z' can return indeterminate only within the proper remit of the test (when v==.z is True)
since 'sysmiss & O' correctly returns False within validly.

4 Within a command, the sequence of options ifnot, else, when is immaterial; for human readers they are probably most
intelligible written in their 'natural' sequence,
as above.

+-+

|g| Option when(expW) can be useful in controlling the range of ifnot and else; it also controls the range of the initial if condition.
We saw it deployed in the last example. A further example:
Suppose, let us pretend, we have a devised a happiness scale which is computed differently for the married and unmarried,
and within that differently for men and women. If we were to do something like:

.validly happ = scaleMW if married & female

that would be fine, but handles only one eventuality; and if we added an ifnot(x) option, x would be applied
both to all the unmarried and to all males, which fails to capture our intention. But we could do:

.validly gen hap = scMW if married, ifnot(scNW) else ((scMW+scNW)/2) when(female)

which calculates scaleMW only for married women, scaleNW only for non-married women,
and sets the average of the two scales for women of indeterminate marital status. Similarly for males:

.validly happ = scMM if married, ifnot(scNM) else ((scMM+scNM)/2) when(!female)

The sequence sets those of indeterminate gender to sysmiss for variable happ.

This option when used as in this example, is a convenience merely, but it does much simplify coding.
Since when focuses on explicit treatment of conditions, its use is deemed incompatible with option widely.

+-+

|h| On the relation between source and source(r). Consider:

.validly v =(p & q)|t, source

applied to a dataset with the following characteristics:

There are substantial missing-values in p and q and t. But t is missing almost exclusively when
p&q is true (so, given the definition of disjunction,
its missing values do not affect the result). Further, p has noticeably more missing values than q, and almost all of q's missing-values
occur when p is also missing.

THEN the above command, with detail, for such data,
would report:

a noticeable number of .a extended-missing values (from source p),
a few .b (from q), a few .c (from t), and a sizeable chunk of sysmiss.
(These sysmiss coming from p&q when both variables are indeterminate.)

The pattern of .a extended-missing values in result v correctly directs us to
those observations in p which, if we could upgrade only one variable,
would have most impact on our result.

But this may (by declining to adjudicate when two 'sources' of missing are
present as operands to the same operator) mislead on the relevance of the variables.
On the above report, variables q and t look similar; but in the data as described t, even if corrected, would have little relevance;
whereas p and q, if both corrected, would have significant impact.
This may be better seen by an alternative formulation:

.vy y =(p & q)|t, s(r)

which randomly assigns one or other 'source' code when both operands are missing but from different sources. The aim of source(r) is not to assign "reasons for missing"
but to get some sense of relevant (to the expression) "sources of missing".

For the data as here imagined, most of the random allocation will occur between p and q,
so the report would probably give more .a than .b, but a sizeable number of each; with few .c, and no sysmiss.

For some research purposes this (correctly indicating that missing-values in t are here irrelevant,
though numerous in the data) can be more useful than the first report.

The implementation details
of source are described below, and make clear that the missing-values codes for the originating variables
are purely 'as if'; the source variables are not modified in any way.
The notional codes override any extended-missing codes within the data; all missing values within the data are treated
as if they carried the associated notional code of their variable.

In particular, this entails that any
explicit extended-missing references
are interpreted as referring to these notional codes,
not to any extended-missing codes recorded in the data (validly prints a warning
in those cases, but not an error, since a careful use might, just possibly, have been intended).

Note that source differs from Stata's useful misstable in that it explores missing values
in relation to their effect on a specific generate/replace command.

+-+

|i| On the utility of the variant forms source(.x) and source(.x,r).
Standard source is most interpretable when used with generate.
As mentioned, care has to be taken when combining source with replace, since the result variable may already
contain extended-missing values not derived from source.
Consider:

.vy gen v = (p & q), source(r)

.vy rep v = (r|s) if u, source(r)

The first command will use codes .a and .b to flag indeterminacy originating with p and q respectively.
But the second command will use these same codes to refer to sources r and s.
Which would render the pattern of extended-missing values in v impossible to interpret.
So here we do:

.vy gen v = (p & q), s(r)

.vy rep v = r|s if u, s(.f,r)

which ensures that the fresh variables have their sourced missing values identified by a sequence beginning with .f
(the problem, and hence this solution, can arise even with a replace accessing the same variables,
since the values are assigned in the order in which variables are encountered, which may vary).

+-+

|j| This is a further example of the use of validly global.
Suppose that, for some esoteric but defensible analytic purpose, we wished to define a function (to pass to some command other than generate/replace) which took the value p&q
when r was true, and p|q when r was false. One strategy would, obviously, be to construct a corresponding variable, using:

.validly gen fn = p&q if r, ifnot(p|q)

and deploy that as appropriate. Another strategy, keeping with the function requirement, would be to consider using the branching function:

cond(r, p&q, p|q, .)

In that straightforward form the device would not work, because we would fall foul of Stata's mishandling
of relations. But instead we could do:

.validly global paq p&q

.validly global pvq p|q

and then use, correctly, the function:

cond(r, $paq, $pvq, .)

wherever it is required. The strategy of "define macro and deploy" extends readily to any embedded logical and relational expression anywhere within Stata, to yield valid answers.

+-+

|k| validly assert is correct (as against unvarnished Stata) in its valuation of logical and relational expressions.
But beyond that, there is no "correct" specification of which of the options widely or possible should be deployed.

For example: We rightly expect "age" to exceed "years of education". If we wished to test whether our sample contained any persons for whom their recorded years of education in fact
exceeded or equalled their recorded age, we could do::

.validly assert age > ed_years, possible

On this setting, if the relation is evaluated and deemed to be indeterminate we do not report a
contradiction (for that observation the relation might possibly,
were the true values known, be true). Only if it is known
for some observation that ed_years>=Age would this call report a contradiction.
Contrast:

.validly assert age > ed_years

which asserts that the
relation is known to be true (so sees any observations for which
the relation is indeterminate as contradicting that
claim to knowledge).

This particular example could readily be handled within standard Stata by use of mi(); in more complex conditional assertions
validly assert is more coherent, and clearer, than unvarnished assert, but whether it is better with or without
possible or widely, depends on your substantive concerns.

For some assertions a combination of validly and unvarnished Stata may be best.
For example, suppose we wish to test whether 'p|q' and 'r|s' have identical patterns of T, F and missing values.
Neither of:

.assert (p|q)==(r|s)

.validly assert (p|q)==(r|s)

do what is required (and adding possible does not help).
We should here do (using the strategy of the previous example):

.vy global pvq p|q

.vy global rvs r|s // to get the logic right

.assert $pvq == $rvs // to get literal equivalence right

+-+

|l| Though validly global uses the quotation-less version of global syntax (so no compound quotes), it is nevertheless happy with embedded string variables. For example:

.validly global both (att1=="agree" | att2 =="agree")

Note: whereas plain global swallows anything, validly global is designed to handle
only expressions, so must reference a legitimate one, in the form exp or if exp.

+-+

|m| For explicit invocations of generate the standard Stata modifiers are available; for example:

.label define tf 1 True 0 False

.vy gen byte pVq:tf = p|q

would validly generate pVq as type byte, attaching to it the value labels just defined in tf.
The ability to attach value-labels on-the-fly is useful.
Had we not pre-defined tf, validly would have printed a warning (lest we had misspecified) but would accept the request,
allowing subsequent definition.

However, although [type] is a standard option for Stata's generate it is dangerous;
Stata gives no warning should there be any entailed truncation.
If storage is a concern, better to leave defaults in place and periodically invoke compress; as the Stata help-file observes:
"compress never makes a mistake, results in loss of precision, or hacks off strings".

Explicit vy replace, again following Stata,
allows option nopromote; although of esoteric utility,
this option is less dangerous than type since it reports any entailed truncation.

+-+

|n| This is NOT a validly example, but a footnote on a related issue in the functions max and min,
which validly does not address. For Stata, both max and min
return determinate values unless all the arguments are indeterminate.
This (though well defined) may not always be appropriate.
Suppose (let us pretend) that for our analytic purposes it would make sense to modify the resource variable, for
our dataset on couples, to record for outright homeowners the higher of the individual incomes,
or missing if that were undefined,
and thought to do:

.replace resource = max(incF,incM) if own==1

For women whose spouses (perhaps bashful tycoons or shamefaced paupers) refused to answer, this returns the income of the woman as the purportedly known higher individual income.
Here better use the assistance of cond to do::

.replace resource = cond(mi(incF,incM), ., max(incF,incM)) if own==1

or, more ingeniously, but less transparently:

.replace resource = cond(incF>incM, incF, incM) if own==1

Either appropriately sets resource as unknown for couples where we lack the relevant information.
The second, ingenious, strategy is lifted from Kantor and Cox (2005)
(but note that,
if differing extended-missing values are present on the two variables,
it always chooses to passthru the alphabetically 'higher').

Note that placing an explicit !mi(incF,incM) test in the conditional would not here
give the same result. Note further that validly, if presented with min or max,
does not intervene to help, but does print a reminder of the issues.

+

| goto Index
+

Method

[This Section (with the next) provides a description of the backstage mechanics underlying the results.
Those wishing simply to use the utility might perhaps skim these sections, but should have no immediate need for careful perusal.
The next useful-for-use section is Utilities, below]

For unvarnished Stata, as discussed, no combination of logical or relational operators returns missing values whatever
the status of their operands. This presents problems, requiring non-transparent, multi-statement, workarounds --
so that missing values are painted by the non-filled-in observations.

A solution would be to create valid functions (but Stata allows users to invent programs, not functions, other than for egen).
Stata's cond function can be made to serve, but rapidly becomes too complex for hand coding.

validly takes logical or relational expressions, converts them to RPN, and generates,
and then executes, the nested cond functions
(to a serious degree of complexity) to deliver the result which the expression requires.

RPN: Whereas standard mathematical notation writes operators between operands, invoking precedence rules to determine sequence,
and additionally using brackets to delimit scope,
Reversed Polish Notation (RPN) places operators after operands, and requires no brackets or precedence rules.
The additional charm of RPN is that, once we have re-formed the expression, we have a formulation which when
read left-to-right is effectively being read in the correct evaluation sequence, so mechanical evaluation becomes straightforward to implement. Thus, for example, consider:

.generate y = (p&q)|!r

The assigned expression becomes, in RPN:

p q & r ! |

Add one further conceptual element: an imagined "push-down/ pop-up" stack (think, pace NRA, "rifle magazine").
Then we can read along the RPN expression: placing operands, as we encounter them, into this push-down stack; applying operators,
as we encounter them, to the top two cells of the stack (or the top cell, for unary operators), allowing the stack to pop-up.
Add a last contribution:
valid functions coding these operators. We define them below, but for the moment suppose the valid function for conjunction
were "and(p,q)", for negation were
"not(p)", for disjunction were "or(p,q)". Coding of (p&q)|!r into a valid function can now be done, entirely "mechanically"

So, taking our example expression -- in RPN form: p q & r ! | -- we put p, and then q, in the push-down stack, giving:

| q |

| p |

+-----+

We then encounter the conjunction operator &, apply its valid function to the top two cells, and combine them,
so the stack now contains only 'and(p,q)', in its top cell.
We next, as we scan our RPN, encounter operand r, so add it to the push-down stack, giving:

| r |

| and(p,q) |

+
+

Next we encounter the negation operator !, so we apply it, modifying the top cell:

| not(r) |

| and(p,q) |

+
+

The final item is the disjunction operator. Applying its valid function to the top two cells of the
stack we would get 'or(cell2, cell1)',
to be filled in by the cell values above. The two cells
are now combined (by this function) into the one cell.
So we have exhausted our RPN expression, we have a one-cell stack, the top cell of which, filled-out, now reads:

| or( and(p,q), not(r) ) |

+
+

Using this, we have:

.generate y = or(and(p,q),not(r))

and that nested function would give the right answer
returning T, F or missing as appropriate.

This was a very simple example, but the procedure can obviously be extended to expressions of any degree of complexity.
Provided the individual functions code validly for T, F or missing the composite nested result will code validly for T, F or missing.
It would just be a mite tedious (and error prone) to do this by hand.

validly handles the translation from standard notation to RPN, and the generation of the resulting nested cond expressions --
as described in the next section.

As use of the detail option will reveal, the final executable code is rebarbative --
and computationally expensive for large data sets. (A Stata redefinition of Stata operators
would make sense, and render 'validly' redundant --
see the article cited below.)

+

| goto Index
+

Functions

[This Section (with the previous one) provides a description of the backstage mechanics underlying the results.
Those wishing simply to use the utility might perhaps skim these sections, but should have no immediate need for careful perusal.
The next useful-for-use section is Utilities, below]

To code logical and relational operators we use the Stata function cond, which tests an expression and returns one of three values depending
upon whether the expression is T, F or missing.
The coding of these logical/relational operators through cond is the 'original' portion of validly. The generation of the RPN, the precursor to applying this coding, is finicky in places,
but has been done many times before.

Note that logical operators are completely subsumed by cond; relational operators appear embedded in calls to cond.
The valid functional forms deployed in place of the given logical and relational operators are:

[In what follows, p and q may be variables, or algebraic expressions, or previously coded logical or relational expressions.]

Negation: !p is coded:

cond(p,0,1,.)

When option extended is set, the function becomes:

cond(p,0,1,p)

Exception: if this is the last-evaluated operator in a condition
which is scheduled to be read as "missing->False" (see below), THEN both codings simplify to:

cond(p,0,1)

Conjunction: p&q is coded:

cond(p,cond(q,1,0,.),0,cond(q,.,0))

When option extended is set, the function becomes:

cond(p,cond(q,1,0,q),0,cond(q,p,0,cond(p==q,p,.)))

Exception: if this is the last-evaluated operator in a condition
which is scheduled to be read as "missing->False" (see below), THEN both codings simplify to:

cond(p,cond(q,1,0,0),0,0)

Disjunction: p|q is coded:

cond(p,1,cond(q,1,0,.),cond(q,1,.,.))

When option extended is set, the function becomes:

cond(p,1,cond(q,1,0,q),cond(q,1,p,cond(p==q,p,.)))

Exception: if this is the last-evaluated operator in a condition
which is scheduled to be read as "missing->False" (see below), THEN both codings simplify to:

cond(p,1,cond(q,1,0,0),cond(q,1,0,0))

Relation: pRq (where R is one of: == != > >= < <=) is coded:

cond(p<.&q<.,pRq,.)

or, if the operands are of type string:

cond(p!=""&q!="",pRq,.)

When option extended is set, the function becomes:

cond(p<.,cond(q<.,pRq,q),cond(p==q|q<.,p,.))

or, if the operands are of type string:

cond(p!=""&q!="",pRq,.s)

Note1: Though it would be more elegant to use mi(), in place of the explicit segregated missing-value checks, this is not viable, since in certain circumstances (such as relations of relations)
that would generate nested calls to mi(), which Stata disallows.

Note2: Special provision is made if an operand is an explicit missing-value.

Exception: if this is the last-evaluated operator in a condition
which is scheduled to be read as "missing->False" (see below), THEN these codings become:

cond(p<.&q<.,pRq,0)

or, if the operands are of type string:

cond(p!=""&q!="",pRq,0)

CONSIDER:

.validly generate v = exp if expC

which takes exp, and by recursive calls to the above functions as required, generates a valid functional form, in local macro `fn', which can return T or F
or indeterminate. It then does the same for expC into function `fnC'.
Next it considers the condition:

Condition: condition "if `fnC' " is, in terms of logic, coded, to set missing to False:

if cond(`fnC',1,0,0) (expressed as "if `f2_mF' " )

to coerce Stata to include only when known-true.

(The actual coding may differ from the above since validly
will, for compactness, modify the coding of any top-level
relation incorporated in that condition, to achieve the exact same effect; these codings are detailed under Exception, above.)

When option widely is set, the expression remains:

if `fnC'

EXECUTION

validly executes the Stata command:

.generate v = `fn' if `fnC_mF'

The exact same logic applies to validly replace

When widely is set, the command becomes:

.generate v = `fn' if `fnC'

IFNOT, ELSE, WHEN

If any of ifnot(expN),
else(expE), when(expW)
are specified the logic remains the same, though the computational mechanics are slightly different.
The program takes the expressions -- expN, expE, expW -- and turns them into valid functions --
`fnN', `fnE' -- using the coding strategies above. Since expW is specifying a condition it is coded into `fnW_mF' (as above) in "missing->False" form
(to ensure correct interpretation by Stata). Function `fnC', and function `fnW_mF' if needed, are then formed into workspace variables:

.generate zvarC = `fnC'

.generate zvarW = `fnW_mF' // if needed

The workspace variables are introduced to avoid repeated calculation of CPU-intensive functions. The program then, for an ifnot…else expression, issues the raw Stata commands:

.generate v = `fn' if zvarC==1

.replace v = `fnN' if zvarC==0

.replace v = `fnE' if zvarC>=.

If the constraint when(expW) has been specified, each condition above is modified by a test for 'expW True':

.replace v = `fn' if zvarC==1 & zvarW

.replace v = `fnN' if zvarC==0 & zvarW

.replace v = `fnE' if zvarC>=. & zvarW

(Had we stored `fnW' in zvarW these conditions would have been, for example:

.replace v = `fn' if zvarC==1 & zvarW==1

but storing `fnW_mf' yields more compact code.)

A logically equivalent strategy for interpreting … if … ifnot … else, would be:

.replace|generate v = cond(`fnC',`fn',`fnN',`fnE')

whilst if…ifnot in the absence of else would be

.replace v = cond(`fnC',`fn',`fnN',v)

and in the absence of both would be:

.replace v = cond(`fnC',`fn',v,v)

for generate, these embedded references to v would be replaced by
sysmiss.

This logically equivalent strategy, though conceptually neater (using the conditional strength of cond), does not make practical sense – the resulting single cond
invoking four major functions would have a higher chance of tripping Stata's size and complexity constraints than
would the adopted 'construct workspace variables' implementation.

Notes

1 The hypothetical extended-missing values invoked by source are handled thusly:
Option extended is switched on (but, because of the next step, all extended-missing values currently in the data will nevertheless be overridden).
Next, suppose p is the first-encountered variable,
having missing values,
in the RPN expansions; in the functions above it is logically as if it were replaced by cond(p,p,0,.a)
-- so ignoring its own extended-missing codes (if any) and treating all its missing values 'as if' having the .a code.
Suppose r is the next-encountered variable with missing values; it is replaced by cond(r,r,0,.b), and so on
(the code already allocated to indeterminacy in string variables is preserved).
The functions are then executed as before.

Under source(r), the
random splitting of sysmiss
(which otherwise flags a conflict of extended-missing values)
is handled
by replacing the last sysmiss symbol (the last argument) in
the extended definitions of conjunction, disjunction and relation (above) by:

cond(int(2*runiform()),p,q)

which returns p-missing-value or q-missing value, each with .5 probability.

The text reads "logically as if it were replaced", since technically validly keeps track of the component parts of the 'sourced' variable,
allowing various simplifications in the coding; for example, where p and q are variables, 'p&q' under s(r) yields:

cond(p,cond(q,1,0,.b),0,cond(q,.a,0,cond(int(2*runiform()),.a ,.b)))

which, though certainly not short, is much shorter than straight substitution would yield.

2 As discussed, validly generate … if … , extended will retain
extended "reasons" for missing values attributable
to the indeterminacy of the conditional, thusly:
The "workspace" strategy is invoked, `fn' is stored in zvar, and, after the first .generate, we have the further command:

.replace v = zvarC if zvarC>. & (zvar<. | zvar==zvarC)

The complex condition is needed to ensure that "two different 'reasons' for missing" is recorded as sysmiss
and not arbitrarily assigned to one 'reason'.
(Under source(r) that sysmiss is randomly split.)

3 Explicit missing values in relations. As discussed, these are coded literally; with the exception of an equality ('==')
involving an explicit extended-missing value (e.g. 'p==.x').
That is coded:

cond(p==. ,. ,p==.x )

to capture a valid indeterminacy (though in 'missing->False' form we revert to literal 'p==.x').

4 Efficiency: In the conjunction and disjunction coding above, when q is known to have values 1,0,sysmiss only, the argument 'cond(q,1,0,.)'
in the second or third position, simplifies to 'q'.

When option extended is set, and q is
known to have values 1,0 or generic missing, the argument 'cond(q,1,0,q)' in conjunction and disjunction simplifies to 'q'.

The program examines incoming variables, and keeps track of expressions, to allow such substitutions.

5 Further on efficiency: if neither operand contains missing values,
the functions reduce to their plain Stata forms.
If one operand has no missing values, other simplifications are possible.
For example, in Conjunction above, if p has no missing values,
the code can be simplified to cond(p,cond(q,1,0,.),0)

The program examines incoming variables, and keeps track of expressions, to allow such substitutions.

But note, in relation to validly global,
that the resulting macros optimised are only useable if the pattern of no-missing is retained;
if applied to data with fresh missing values the macros will yield invalid results;
validly issues a reminder (the optimisation can be disabled).

6 Yet more on efficiency: since & and | are commutative, and pR1q can be readily rewritten as qR2p,
at each particular coding there is a choice of whether to code for, say, p&q or q&p, depending on the current contents of 'p' and 'q'.
The program chooses, to minimise the number of calls to cond, and, if two formulations are equivalent, minimises bracket depth
(and reports any move from the default sequence by talking of "valid optimised function" when detail is set).
(Should you wish more readily to "see" the RPN->cond coding this optimisation can be switched off.)

7 And a final note on efficiency: The above functions (particularly the more complex) can be hard coded in a variety of different ways.
For example, we could also code Relation/extended above, as, variously:

cond(p>=.|q>=.,cond(p>=.&q>=.,cond(p==q,p,.),cond(p,q,q,p)),pRq) or

cond(p<.,cond(q<.,pRq,q),cond(q<.,p,cond(p==q,p,.)))

In deciding which to deploy, since these are for machine generation, 'concise' has been rated above 'transparent'.

8 detail as an option, will display the RPN and the
nested cond functions, and provide helpful commentary on what is being done.

9 The commands validly generate/replace/assert
also, whatever the option, return the RPN code in macro r(RPN); the variant validly conditional_command always returns the full constructed condition in r(ifcond).

10 Extended-missing within algebra (see earlier comment):
this is a footnote, not about validly, but about a related topic.

You could hand-code any individual dyadic algebraic
operator to retain extended-missing values by using the Relation/extended coding above,
but now replacing relation R by the appropriate algebraic operator.

Further, a single variable function, fn(p), could be handled by
cond(p<.,fn(p),p) or equivalently cond(mi(p),p,fn(p)) (note that the test is here on 'p', not 'fn(p)',
since we seek to isolate "undefined because argument-undefined" in contrast to, say, "undefined because log-of-negative").
Unary minus could be seen as a unary function, giving cond(p<.,-p,p),
or alternatively cond(p,-p,0,p).

The only feasible way to apply this to
even moderately complex algebra would be to sequentially construct temporary variables for each individual operation.

So, for example, '.gen v = (p+q)^r' becomes:

.gen pq = cond( p<.,cond(q<., p+q,q),cond( p==q|q<., p,.))

.gen v = cond(pq<.,cond(r<.,pq^r,r),cond(pq==r|r<.,pq,.))

The more complex, the more tedious.

If much of this were desired, there would be merit,
for binary operators, in splitting that function into two macros, thus:

.local m1 cond(\`v1'<.,cond(\`v2'<.,\`v1'

.local m2 \`v2',\`v2'),cond(\`v1'==\`v2'|\`v2'<.,\`v1',.))

(where the backslash inhibits expansion, at point of definition, of the embedded macros)
and then filling-in and invoking the two main macros repeatedly. Thus the above would become:

.local v1 p // set embedded vars

.local v2 q

.gen pq = `m1' + `m2' // macros with algebraic binary operator

.local v1 pq // set embedded vars

.local v2 r

.gen v = `m1' ^ `m2' // macros with algebraic binary operator

Still tedious, but can be repeated (stepwise, from binary operators, to build complex expressions) with less chance of error.

+

| goto Index
+

Utility to generate test variables

To allow ready exploration of the implementation of logical and relational expressions, there is a utility:

.validly gen_test_vars newvarlist [, extended nonind ]

which generates between two and ten fresh indicator variables, forming, for these variables, all possible patterns involving the three values: 1, 0 and sysmiss ---
so there are var^3 patterns.
Whence, rather than checking your coding by searching across your dataset for patterns of values, you could generate a compact
bespoke dataset, with similar variable names, embodying all possible patterns, and check coding on that.

If extended is requested, the patterns extend across five values: 1, 0, sysmiss, .a and .b (to allow exploration of transmission of extended-missing values).
But note that there are now var^5 patterns, so number of variables is restricted to six.
(This command does not attend to whether set_extended is on, only to its own explicit extended option.)

If nonind is requested, the variables constructed are not indicator variables; observations
which on the default setting would have value '1' on a variable, now carry random integers in the range 1-100.
Given the way Stata interprets non-zero as True, this gives the exact-same pattern of True, False,
missing as the default setting. It is included to allow checking that validly and/or your coding works correctly on
non-indicator, as well as indicator, variables.

A trivial example:

.vy gen_test_vars p q

would give the nine possible patterns for p and q displayed in the first two columns of the expository table above.

This utility is also used to generate the test data, containing all possible combinations of values, on which the program's validation file runs.

Minor Utilities

Teetering on the edge of Stata programming propriety, validly uses the global macro $validlyDefault
to store the following default settings.
They are thus retained, across calls to validly, within the one Stata session.

(Checks are
made that this specific global has not, implausibly, been used by some other process; if it has, user is given choice of action.)

validly set_extended [on|off] explicitly
sets extended switch to on or off. Default is off;
bear in mind the costs of setting this to ON.
When the default is on, commands (such as the generation of a string variable) for which the
extended option would be inappropriate will nevertheless continue to work
(the setting is suspended for the duration of the command).

validly set_detail [on|off] explicitly sets
detail to on or off (default is off).
If your preferred mode is detailed reporting, it makes sense to set
this to 'on' at the start a Stata session, rather than repeatedly specifying option detail.

validly set_reminder [on|off] controls the switch for reminders
when unfamiliar operator precedence in unbracketed
expressions overrides an 'intuitive' left-to-right reading
(default is on).

validly set_compact [on|off] sets
one optimisation strategy
to on or off (default is on).

validly set_nomiss [on|off] sets
optimisation for variables with no missing values
to on or off (default is on).

validly set_impute [on|off] explicitly
sets the ability to impute replace to on or off (default is
on).
The risk of confusion is small; but some may feel more secure with this turned off.

validly set_nullstring .x
where .x is a specific extended-missing value,
overrides the default .s value used when, with extended set, a null-string is encountered in a relation.
(Note that validly ensures that whatever value you choose does not interfere with the coding of option source.)

validly set_defaults returns all of the above
switches to their default settings.

Validation File

The program has been extensively tested in use, but a more formal test file:

validate_validly.do

accompanies this package. If loaded and run, this .do file will, with commentary, exercise validly,
on a data set built by gen_test_vars.
The data contains all possible patterns of values across a group of variables, so constructs can be exhaustively tested.
For example, each logical or relational operator is first validated in a bivariate setting.
These verified bivariate relations can then be used, via intermediate variables, to construct verified complex expressions.
The displayed commentary as the file is run is reasonably informative on the tests being performed,
but since the commands (being many, looping) are run in quiet mode,
for detailed specification the content of the .do file should be directly inspected.
The file is supplied to give some indication of the extent of testing, and so reassurance of robustness.

+

| goto Index
+

Two restrictions

no arithmetic allowed on the results of logical/relational expressions

Because of the
method it uses for parsing (on logical and relational operators), validly
refuses to handle expressions which
attempt arithmetic upon the results of logical and relational expressions. So, for example, (p>q)+4 fails (with advice).

These would be substantively odd expressions (though admittedly definable, it is a mite odd
to do arithmetic on things which are True or False), so in real life this restriction is unlikely to constitute
a substantial impediment to progress. If however you have an
ineluctable need, the strategy is to validly generate variables corresponding
to the logical or relational expressions, and then carry out arithmetic on these constructed variables. Alternatively,
for logical expressions, use validly
global to encode into valid functions, embed the macro(s) appropriately
and reissue the command (the macro route does not work for relational expressions
because of the next restriction).

logical/relational expressions within functions are not parsed

This, for which it is also difficult to think of plausible instances, is again a consequence of the approach that validly
takes to parsing; thus 'max(p&q,r&s)==1' will be rejected (with advice).

The workaround is to validly generate variables corresponding
to the logical or relational expressions, and reissue the command with these variables, in place of the expressions, within functions. Alternatively,
for logical expressions, use validly
global to encode into valid functions, embed the macro(s) appropriately
and reissue the command (the macro route does not work for relational expressions
since their coding, see above, itself involves relational expressions).

Example, consider a rejected:

.validly y = max(p&q,r) if s|!r

Instead we could do:

.validly gen y1 = p&q

.validly y = max(y1,r) if s|!r

or, equivalently, we could do:

.validly global y1 p&q

.validly y = max($y1,r) if s|!r

BUT note that max itself has further problems with missing-values - see the footnote above.

+

| goto Index
+

Error messages

invalid syntax -- and related error messages -- probably does point to some error you have made, though you should check for misspellings, invalid variables,
invalid options, as well as "invalid syntax", more conventionally understood
--- as with any syntax checker,
the message may not always point to the precipitating error.

Suggestion if the error continues to
elude you: sometimes Stata's syntax checker if fed the plain command (stripped of "validly", and of course validly's options)
may be more explicit.

Note: validly first looks for generate/replace/assert/global,
next it looks for a syntactically correct imputed generate/replace,
and a wrapped command is assumed when these are not found.
This, in particular, means that an intended imputed generate/replace which is syntactically misspecified
can lead to an error report appearing to mutter about an invalid perceived wrapped command.

No syntax checker can itself check that what you have written
(especially where exp is at all complex) matches your intent. So. It is always good practice after any moderately complex
construction to list the constructing
variables and the result variable across an appropriate small sample of observations (using in, or perhaps if)
and check that your intent has been instantiated.
The included utility to generate test variables may simplify the task of checking across all-possible patterns of values.

Be alert to the ways in which operator precedence may interpret an expression as you did not intend. Stata follows
standard precedence rules
amongst arithmetic, relational and logical operators; validly does the same. Whilst most of us "naturally" (i.e. through early habituation) read 4+2*3
as 4+(2*3), our intuition that 'p|q & r' is properly read as 'p | (q&r)' may well be weaker.
Or, consider:

p&q == r&s

Stata, and validly also, again following standard precedence rules, will interpret this as:

p & (q==r) & s

which may well not match the writer's intention. So, when dealing with logical and relational operators,
the advice is "use brackets to clarify intention"; explicit brackets always
dominate implicit precedence rules. Excess of brackets does not do harm; absence may.

As a minor crutch, validly issues a reminder
if either of the less-familiar precedence rules ("relations take precedence over logical operators",
"conjunction takes precedence over disjunction") has, in the absence of brackets, overridden an
'intuitive' left->right reading of an expression. Thus, the expression
'p|q&r' attracts comment but the equivalent 'q&r|p' does not,
and nor does 'p|(q&r)'.
The RPN versions (revealed through use of
detail, or by typing plain vy)
show unambiguously how your expressions have been understood.
It should be stressed that validly's parsing is here identical to plain Stata (or indeed any mathematical program); it just adds some monitoring.
If you find such nannying annoying it can be suppressed for the duration of a session

The most insidious errors are those where a program merrily, without complaining, executes a command, but the effect of that command is not what was intended.
This is why (as the discussion partly explains) Stata's default conventions are unsatisfactory. The untutored user may mistakenly write:

.generate pvq = p|q

But, more worryingly, the mid-range user, alerted to the need to "attend to missing values", may write:

.generate pvq = p|q if !mi(p,q)

but be equally mistaken – and, more dangerously, nothing will alert such a user to the error. Which is the case for writing, accurately and, importantly, transparently:

.validly generate pvq = p|q

expression too long or, more rarely, too many macros
may be encountered when extended is set and the raw expressions are very complex; it indicates that the generating functions
are too complex or nested for Stata. The solution is
EITHER to decide that keeping track of extended-missing values was not so important after all and drop the extended option
OR to use validly to generate variables corresponding to subexpressions. For example:

.validly generate v = !((p&q)|(p|q))&(r|!s)&t , extended

will be rejected. But we can do (remembering to use the e option):

.validly generate paq = p&q, e

.validly generate pvq = p|q, e

at which point:

.validly generate v = !((paq)|(pvq))&(r|!s)&t , e

will execute quite happily. The problem arises only for unusually complex expressions where you are trying to preserve extended-missing values
(remember that source also invokes this mode, though more parsimonious in its coding).

error coloured text sometimes is used (not extensively) in the output from validly to flag, not errors, but
things-you-should notice (this because that colour is the one most likely to be consistently noticeable across varying colour-schemes).
An example being text noting when the implied action is set to replace.
The output from validly may also contain some text coloured as links -- these are indeed links to portions of this help file.
(Parenthetically, this help file is most legible if you avoid the one standard colour scheme -- edit/preferences/ viewer_colours/studio -- which underlines links.)

Error messages should all be ascribable to your syntax errors
(widely understood). Should they appear not to be, contact the author with some urgency (please!);
the program has been extensively tested -- and
a validation file accompanies this package.

+

| goto Index
+

Saved results

The local macros discussed in detail reporting mode (such as `fnC') are of course internal to validly.
But all invocations of validly return the following macros:

r(RPN) which contains the generated RPN.

r(precedence) which contains flags, for each expression, reporting whether precedence has been invoked.

[to display: The information in both these macros can be displayed by reissuing the command .vy with no arguments. Thus suppose we had issued:

.vy v = p&q == r&s

which would yield a reminder about '==' taking precedence over '&'. Suppose we were puzzled. The command:

.vy

would display the RPN, confirming that, correctly, this has been read as '(p&q)==(r&s)'.
Targeted use of argument-free .vy may give most of the utility of detail with less chatter.

Additionally, wrapped commands return the following macro:

r(ifcond) which carries the condition deployed in validly running the wrapped command, and so gives,
in executable 'cond' form,
the generated valid conditional function;
this can be extracted and retained by, say:

.global filter `r(ifcond)'

and this global macro, $filter, could then be used to repeat the condition
(though not within a fresh call to validly).
(If the condition has been specifically optimised to match the absence of missing data in some variables, then, to avoid future error, r(ifcond) is WITHELD.)
Click here for an example of correct macro use.

These wrapped commands also, if appropriate, return their own stored:

r() or e() these saved
results are defined exactly as they would be were the wrapped command issued 'raw'.
Should the wrapped command be an estimation command, then the command line issued by validly
will be stored in e(cmdline)

Displaying saved results: For r(RPN) the best strategy (as described above) is to type vy (or validly)
with no arguments -- this displays the stored information elegantly.
Other arguments can be displayed directly. So, for example:

.di e(cmdline)

gives the command line issued by validly for a wrapped command.

Should e(cmdline) happen to be too long to display simply, the "quoted-macro" form:

.di `"`e(cmdline)'"'

will always work. If the command itself contains no quotation marks, the compound double quotes
`"xxx"' can be replaced by the simple, "xxx", form.

Users should ignore the global macro $validlyDefault which is retained by validly as a 'memory' for settings between calls,
within one Stata session.

References

Article: Missing values in relational and logical expressions in Stata; a problem and two solutions.

Kantor, D & Cox, N J (2005) Depending on conditions: a tutorial on the
cond() function. The Stata Journal, 5, p413-420

Author's email:

Comment, suggestions, welcomed; an error report would be unwelcome -- but shall be attended to.

kenneth.macdonald @ nuffield.ox.ac.uk

Also see

Online: [D] generate [U] operators [U] missing

+

| goto Index
+

help for validly v3.2.1 13/8/13