# RSTEST: help and advice

Consult the reference manual for further details and worked examples.
W.G.Bardsley, University of Manchester, U.K.

Concerning signs and runs

Signs can arise as differences between matched sample values, or as symbols for, e.g. success or failure, or better or worse. The sign test explores whether an excess of either type of sign suggests departure from

         H0: Both signs are equally likely.

A set of signed numbers also gives rise to a runs pattern. For instance, a set of differences (e.g. residuals) might be:

         0.1, 0.8, 0.2, -1.2, -0.8, 0.7, -1.1, -0.4, 0.2, 0.6, -0.3

which is equivalent to the sign pattern:

             +   +   +   -   -   +   -   -   +   +   -

Here there are 11 residuals, 5 negatives, 6 positives, and 6 runs (i.e. groups of like signs). A runs test can be used to test the null hypothesis

         H0: Signs are uncorrelated.

In sequences of supposedly independent random numbers, long sub-sequences of successively increasing or decreasing values suggest lack of independence. The runs up and down test is used to test

        H0: The values are uncorrelated.

The sign test

Suppose that there are m results with one sign and n of the opposite sign. (Signs can be + = success, - = failure, etc).
If S (=< m + n) is any possible number of signs, the program calculates the two tail sign test statistic TS defined by

         TS = 2.0*P(S =< min(m,n)) (or TS = 1 if m = n).

For a significance test of size alpha, e.g. alpha = .05 and the null hypothesis

         H0: The signs are random,

consider rejecting H0 if TS < alpha against the alternative

         H1: One of the signs predominates.

To perform a sign test you can type in m and n from a terminal, or input matched pairs of samples to be differenced in order to calculate lower and upper tail sign statistics, and the two-tailed sign test statistic. These can be used to test the one sided alternative hypothesis

         H1: Just one of the signs predominates.

The run test

Suppose there are m residuals of one sign and n of opposite sign, and k runs occur in the sequence of residuals. If R is any possible number of runs, the program calculates the lower tail run test statistic defined as

         P(R =< k; given m and n),

and the (less powerful) lower tail total run test statistic

         P(R =< k; given m plus n).

An exact method is used, unless m > 20 and n > 20, when the normal distribution approximation is used.

For a significance test of size alpha, e.g. alpha = .05, and

         H0: The residuals are uncorrelated and of random sign,

consider rejecting H0 if P(R =< k) < alpha, against

         H1: The residuals are correlated (i.e. have trends).

To calculate upper tail probabilities just input m, n and k - 1 from the terminal since

         P(upper tail, k) = P(R >= k)
                          = 1.0 - P(R =< (k - 1)))

Using sign and run tests to analyse trends in residuals

You should consider rejecting a model at the 1% level if the probability of (runs =< the number observed) is < .01, i.e. the observed number of runs is =< the 1% critical level. Similarly, consider rejecting a model at the 5% level if the probability of (runs =< the number observed) is < .05, i.e. the observed number of runs is =< the 5% critical level.

Note that warnings are issued if the program detects that inconsistent data have been supplied, and you must correct the mistake and try again.

The program will calculate the probability of runs less than or equal to the number observed(total lower tail probability conditional upon the number of non-zero residuals) and also the probability of a number of signs less than or equal to the minimum number observed (two tail sign statistic). These are less powerful than the previous statistic, but they can sometimes be very useful.

The runs up or down test for randomness

This test should only be used with very large samples (e.g. n >= 1000 or preferably much more) that have no tied values. It estimates the probability of monotonically increasing runs up or monotonically decreasing runs down of differing lengths in continous data, where the probability of two numbers having precisely the same value in a random sample is vanishingly small. Typically it is used to test the performance of random number generators.

The Kolmogorov-Smirnov one-sample test

This is used to test whether it is likely that a random sample resembles a sample from a stated distribution. The test can be done for the F, t, Normal and chi-square distributions, using programs FTEST, TTEST, NORMAL or CHISQD, and this program adds tests for Beta, Gamma, Binomial, Exponential, and also Poisson distributions. It is a good test for continuous distributions where the population parameters are known, but is not so good when the parameters have to estimated from the sample.

Kolmogorov-Smirnov two-sample and Mann-Whitney U tests

These approximate tests are used to compare two samples to see whether they come from the same distribution. The tests explore whether one sample tends to have larger values than the other.

Wilcoxon signed rank test

This can be regarded as a nonparametric equivalent of the paired t-test. Of course both samples must be the same size.

The median, Mood and David tests

The median test attempts to establish if two samples have the same medians, while the Mood and David tests are designed to examine the hypothesis of equal dispersion, i.e. identical variances.

All possible pairs

If a modest set of n samples (say < 6, as 5! = 120) are available as vector files referenced in a library file, then all possible n! Mann-Whitney U tests can be done to probe for possible differences between the sample distributions.

Multivariate analysis

For a multivariate sample, both Kendall and Spearman correlation analysis can be done, with the results returned as a matrix of correlation coefficients, and also as a matrix of significance values.

For a set of m samples of arbitrary length a Kruskal-Wallis 1-way ANOVA can be done and, if all samples are of equal size, Friedman 2-way ANOVA can be carried out on the n by m matrix.

To explore connections between m variables measured on n subjects with the results in a n by m matrix, the Kendall coefficient of concordance can be calculated.

In the special case of repeated measures with a matrix of m binary responses for n subjects, the Cochran Q test can be done on the resulting matrix of zeros and ones. If required, the first column can be successive integers to identify the subjects.

Regression on ranks

Where n responses depending on m variables have been measured in k groups, it is possible to perform regression on the ranks of the observations, not the actual values. The data file must be in the form of a n by m + 3 matrix in which columns 1 to m are the variables, column m + 1 is an indicator variable for censoring, column m + 2 contains the observations, and column m + 3 has the stratum indicators, i.e., the group numbers. If all observations were measured in some range, then all censoring variables are set to 0 and the supposed error distribution can be selected. If some observations were capped at limiting values, i.e. the true values were outside the range of measurement, there is right censoring, and the corresponding censoring indicator must be set to 1. In this case a parameter gamma must be set, and this selects the skew extreme value distribution for small gamma, the symmetric logistic distribution for gamma equal one, and the negative exponential distribution for large gamma.

Advice on which test to use

- If you have data that is equivalent to signs only (e.g. above the median is +, below is -, etc.) just use the sign test. If you think the order of signs is also important do a run and sign test. You can do this by
  (a) typing in the no. of signs and runs,
  (b) reading in a set of of signed numbers(e.g. residuals), or
  (c) by reading in two paired samples for differencing.
- With large samples, the runs up or down test detects sub-sequences of improbably long runs with increasing or decreasing values.
- If you want to see if your data could be a random sample from a known distribution, use the Kolmogorov-Smirnov 1 sample test.
- If you have two independent samples and you just want to see if they could both come from the same distribution, you should use the Kolmogorov-Smirnov 2 sample and/or the Mann-Whitney U test.
- Use Spearman/Kendall-tau for non-parametric correlations.
- Use the Kendall coefficient of concordance test only if your data are in the form of ranks, as in the test file g08daf.tf1.
- Use the Cochran Q repeated measures test for cross classified binary data only if your data are in the form of 0 or 1, as in the test file cochranq.tf1.