*! version 1.0 5Jun2024

help for validarcae

Title

validarcae - provides several functionalities for dealing with
codes from the Portuguese Classification of Economic Activities (CAE)

Syntax

validarcae
varname [if], [options]

where varname is either a string or numeric variable containing CAE codes

Options

Options
rev(#) allows the user to specify which CAE Revision to use.
The user may choose between 1, 2, 21, or 3. If the option is not specified it defaults
to Revision 3.

getlevels(numlist[, en force]) returns (a) variable(s)
with the same or higher aggregation level for validated codes.
The levels, as well as their designations,
differ according to CAE revisions and the variables returned reflect those differences.
Below we present the levels for each CAE revision and their designations:

- CAE Rev. 1 : Division (1); Subdivision (2); Class (3); Group (4); Subgroup (5); Split(6)

- CAE Rev. 2 : Section (1); Subsection (2); Division (3); Group (4); Class (5); Subclass(6)

- CAE Rev. 21: Section (1); Subsection (2); Division (3); Group (4); Class (5); Subclass(6)

- CAE Rev. 3 : Section (1); Division (2); Group (3); Class (4); Subclass(5)

The user may select all the levels (1 2 3 4 5 (6)), which will create 5 (6) new variables, depending
on the CAE revision. These newly created variables are numerical and have value labels associated with
each code.

The value label always includes the original code plus the official description. You can
extract the original code (string) from the value label with the option keep.

Observations for which the code is invalid, missing observations, and observations
with a length smaller than the length required for conversion will be coded
as -99 with the value label "Invalid Conversion". Codes with an ambiguous validation
will be coded as -98 with the value label "Ambiguous validation, not able to convert".

en sets English as the language to use for the description of CAE codes included in the value labels.
The default is Portuguese.

force forces the conversion on observations with an ambiguous validation by ignoring the
possibility of leading zeros. force will not work if the ambiguity is generated
because option dropzero was specified.

solve(var[, th en]) this option can
be used if the user has a string variable with the description of
CAE codes. Ambiguous cases are solved
by comparing the description provided by the user with
the official description. The option uses jarowinkler to
select the codes with the closest string distance - the metric is scaled
between 0 (not similar at all) and 1 (exact match).
This option creates variable _solved taking value one if the ambiguity
was solved using the description of the code and zero otherwise.

th is the minimum Jaro-Winkler similarity score allowed to solve ambiguities.
The default is 0.7. If the Jaro-Winkler similarity score between strings is smaller
than this value, the ambiguity will not be solved.

en sets English as the language of the description in the variable provided.
The default is Portuguese.

similarity(var[, en]) computes the Jaro-Winkler similarity metric
between a string variable with the description of CAE codes and the official description,
using jarowinkler - the metric is scaled
between 0 (not similar at all) and 1 (exact match).
This option creates variable _jwsim_rev_#, where # is the Revision number.

en sets English as the language of the description in the variable provided.
The default is Portuguese.

keep creates a variable named _cae_str, which is basically
a string version of the variable provided by the user. However, codes may be
different from those provided by the user, depending on two factors:

(i) option dropzero was specified, effectively changing _cae_str for
codes where there are no ambiguities;

(ii) a 0 was added to the left of codes where _valid_cae_# is equal to 2, 20, 200 or 2000.

fromlabel this option should be used if the variable has value labels and those labels
follow the convention of having the code as the first word. In this case the command validates the codes
that are extracted from the value labels.

IMPORTANT: BPLIM datasets always follow this convention for value labels.

dropzero in some datasets CAE variables mix codes with different levels of aggregation
where codes for higher levels of aggregation are right padded with zeros (and thus are invalid).
This option performs a validation of invalid codes by recursively dropping the rightmost zero from the codes.
The process stops when one of the following conditions is met for every observation:

(i) the code's length is equal to 1;

(ii) the last digit of the code is different from 0.

The option produces the variable _zerosdropped, which contains,
for each observation, the number of zeros that had to be dropped until
one of the aforementioned conditions was met.

The validation codes for these observations are a combination (sum)
of the codes presented below,
because the code is validated every time a zero is dropped.

Description

validarcae is a Stata command that provides several functionalities for
dealing with codes from the Portuguese Classification of Economic Activities (CAE).
The command validates existing codes and descriptions, identifies possible ambiguities,
aggregates codes to higher levels, and provides the official code description in portuguese
and english. The command works with all revisions of the CAE:

- CAE Revision 1 : 1973 - 1993

- CAE Revision 2 : 1994 - 2002

- CAE Revision 21: 2003 - 2007

- CAE Revision 3 : 2008 - ...

Note: the time interval represents the official dates in which a revision
came into force.

The command produces the variable _valid_cae_# signaling valid and invalid codes
for CAE Revision # as specified by the user, where # equals 1, 2, 21 or 3.
Codes from revision 1 have 6 digits (at the highest level of disaggregation) and
always start with a number greater than 0, so the possible values for
_valid_cae_# are:

- 0 : missing obs;

- 1 : valid at 1 digit;

- 10 : valid at 2 digits;

- 100 : valid at 3 digits;

- 1000 : valid at 4 digits;

- 10000 : valid at 5 digits;

- 100000: valid at 6 digits;

- 200000: invalid code.

Codes from revisions 2, 21 and 3 have 5 digits (at the highest level of
disaggregation) and may start with a zero. Zeros on the left are sometimes
inadvertently lost (for example if the codes are converted to numeric format),
so we check if codes with a length smaller than 5 will still be valid
if a 0 is added to the left. Therefore, _valid_cae_# can take on the
following values:

- 0 : missing obs;

- 2 : valid at 2 digits (0 + 1 digit);

- 10 : valid at 2 digits only;

- 20 : valid at 3 digits (0 + 2 digits);

- 30 : valid at 2 digits only or 3 digits (0 + 2 digits);

- 100 : valid at 3 digits only;

- 200 : valid at 4 digits (0 + 3 digits);

- 300 : valid at 3 digits only or 4 digits (0 + 3 digits);

- 1000 : valid at 4 digits only;

- 2000 : valid at 5 digits (0 + 4 digits);

- 3000 : valid at 4 digits only or 5 digits (0 + 4 digits);

- 10000 : valid at 5 digits;

- 200000: invalid code.

Upon conclusion of the command a tabulation of _valid_cae_# is presented.

Examples

Example 1:
Validate CAE codes according to CAE Revision 3.

. validarcae cae

Example 2:
Validate CAE codes according to CAE Revision 1 and create variables rev1_division and
rev1_class with value labels in English.
The new variables contain, respectively, codes for CAE Rev.1 Division and Class.

. validarcae cae, rev(1) getlevels(1 3, en)

Example 3:
Validate CAE codes according to CAE Revision 3 and use variable cae_desc
to solve ambiguous codes. Variable cae_desc contains the Portuguese
description of the CAE codes.

. validarcae cae, solve(cae_desc)

Remarks

Please notice that this software is provided "as is", without warranty of any kind, whether
express, implied, or statutory, including, but not limited to, any warranty of merchantability
or fitness for a particular purpose or any warranty that the contents of the item will be error-free.
In no respect shall the author incur any liability for any damages, including, but limited to,
direct, indirect, special, or consequential damages arising out of, resulting from, or any way
connected to the use of the item, whether or not based upon warranty, contract, tort, or otherwise.

Dependencies

savesome by Nicholas Cox

ustrdist by Michael Barker and Felix Pöge (for option solve)

Author

BPLIM, Banco de Portugal, Portugal.

Email: bplim@bportugal.pt

We appreciate your feedback. Comments are welcome!