Efficiency Considerations Using The SAS System: Rick Langston, SAS Institute Inc., Cary, NC
Efficiency Considerations Using The SAS System: Rick Langston, SAS Institute Inc., Cary, NC
Paper 002-30
ABSTRACT
There are many products being released by SAS Institute that either use an abundance of SAS code, or generate
SAS code, or both. We have begun an internal process to assist product developers in reviewing this SAS code for
issues of efficiency, host compatibility, performance, and so on. We also realize that these issues are of interest and
benefit to all SAS programmers. This paper is a collection of some of our initial performance analyses in writing
efficient SAS code.
INTRODUCTION
This paper examines a few of the most common questions regarding efficient SAS code. Its main focus is to
examine CPU usage differences in the areas listed below. Except where noted, all examples were tested on
production versions of SAS on UNIX platforms.
• IF statements versus WHERE statements in DATA steps
• IF statements versus SELECT statements
• PROC SORT versus SQL ORDER BY
• Issues involving DATA step views
• Functions versus arrays versus direct SAS coding
• DATA step stored programs
• Threaded versus non-threaded performance for PROC SUMMARY
• Table lookup comparing PROC FORMAT, merging, and the hash table object
It is important to understand that the real times and CPU times shown in the logs in this paper are not necessarily
indicative of the times you may see if you try the same tests. These times vary greatly between operating systems
and between machine configurations. The times are used here primarily for purposes of comparison to help you to
make decisions on SAS coding practices.
IF OR WHERE
A common question regarding efficiency is "Which do I use: IF or WHERE?" These examples compare the
performance of three different methods of subsetting a third of the observations in a 500,000-observation data set.
SUBSETTING IF STATEMENT
data temp; set mydata.x500000;
if mod(obs,3)=0;
run;
WHERE CLAUSE
data temp; set mydata.x500000;
where mod(obs,3)=0;
run;
1
SUGI 30 Applications Development
In the three previous examples, the subsetting IF statement is slightly faster, but not significantly so. Interestingly,
the WHERE statement is a little faster than the parenthetical WHERE statement.
Performance in these examples is close enough that the choice of an IF statement versus a WHERE statement
versus a WHERE option is arbitrary. However, if the subsetting is to be performed using a LIBNAME engine against
a database that is optimized for WHERE processing, then the WHERE choice is preferred. A subsetting IF
statement requires all observations to be retrieved from the database, with the DATA step code determining whether
the observation is to be kept. Using a WHERE statement in these circumstances retrieves only those observations
that match the criteria for subsetting.
IF-THEN/ELSE OR SELECT
To generate a significant number of IF-THEN/ELSE statements and SELECT/WHEN statements, we use the
following SAS code to create programs with 100 clauses of each type. The code assigns a random value to y based
on increasing values of x. The file SASCODE1 will contain IF-THEN statements, and the file SASCODE2 will contain
SELECT/WHEN statements. SASCODE1 and SASCODE2 are both TEMP files that are referenced by a %INCLUDE
statement when each DATA step program is run.
IF-THEN/ELSE STATEMENTS
19 data temp; set mydata.x500000;
20 %include sascode1/source2;
21 +if x1=1 then j=0.1093699425 ;
22 +else if x1=2 then j=0.9230094435 ;
23 +else if x1=3 then j=0.7694377665 ;
...
119 +else if x1=99 then j=0.3876921946 ;
120 +else if x1=100 then j=0.9119509677 ;
121 run;
...
NOTE: DATA statement used:
real time 9.17 seconds
cpu time 2.61 seconds
2
SUGI 30 Applications Development
SELECT/WHEN STATEMENTS
122 data temp; set mydata.x500000;
123 %include sascode2/source2;
124 +select(x1);
125 +when(1 )j=0.1093699425 ;
126 +when(2 )j=0.9230094435 ;
...
223 +when(99 )j=0.3876921946 ;
224 +when(100 )j=0.9119509677 ;
225 +otherwise; end;
226 run;
...
NOTE: DATA statement used:
real time 9.08 seconds
cpu time 2.51 seconds
The SAS log shows that 100 IF-THEN/ELSE statements took 2.61 CPU seconds to run, while the corresponding
SELECT statement with 100 WHEN clauses took a little less time, CPU 2.51 seconds.
The SELECT statement is only slightly better in this example. WHEN clauses can have scalar values only, whereas
IF-THEN/ELSE statements don’t have that limitation.
Remember that the ELSE clause makes a big difference in IF-THEN/ELSE performance. If the ELSE clause had
been omitted here, the example would have run much slower. Consider another example:
In the first case, if x=1 and y=2, then it is not possible for x=3 and y=4, so it is appropriate to provide an ELSE
clause. If the ELSE clause were omitted, and the condition of x=1 and y=2 were true, then the execution of the
second IF statement would be unnecessary and cause additional execution time.
The conditions in the two IF statements are not mutually exclusive. Therefore, adding an ELSE clause might cause z
to not be properly set. You must examine your specifications carefully when using IF-THEN/ELSE statements.
Remember also that you can't always substitute a SELECT/WHEN statement for an IF-THEN/ELSE statement. Here
is an example:
This expression cannot be represented using a simple SELECT statement. You could use the following scenario,
which has an implied ELSE clause, but it has no advantage over the IF-THEN/ELSE statement.
select(1);
when(x=1 and y=2) do; z=3; end;
when(x=3 and y=4) do; z=7; end;
end;
SORT OR SQL
This example demonstrates the performance difference between using SQL to sort a data set and using the
traditional PROC SORT approach. Examples show three ways to create a data set TEMP1 with the variables obs
and x3, to sort by those variables, and then to subset.
3
SUGI 30 Applications Development
The execution time is slightly longer for the multi-step process. You might find it faster to combine the subsetting and
sorting into a single step.
Here we perform the same subsetting in PROC SQL using SELECT ... FROM and perform the sorting via the
ORDER BY clause. The execution time is greater, just under eight seconds.
Deciding whether to use PROC SQL depends partly on what engine is involved. SAS/ACCESS engines might be
able to pass SQL code directly to the database for faster execution. PROC SORT can be very competitive in its
sorting techniques. Often you will either be intimately familiar with SAS code and choose PROC SORT, or you will
choose SQL based on that familiarity.
4
SUGI 30 Applications Development
Notice that the overall CPU execution time for the DATA step view example is 3.94 seconds (for the view execution)
+ 1.62 seconds (for the DATA step execution). By contrast, if we create the SAS data set and use a SET statement
on the data set instead of the view, the CPU time is considerably less, only 1.7 seconds. You need to be careful
when using views, especially when dealing with external files. The dynamic nature of external files makes them
powerful but CPU intensive.
SUM FUNCTION
5 data new; set mydata.x500000;
6 sum = sum(of x1-x10);
7 run;
SAS OPERATORS
9 data new; set mydata.x500000;
10 sum = x1+x2+x3+x4+x5+x6+x7+x8+x9+x10;
11 run;
ARRAY STATEMENT
5
SUGI 30 Applications Development
Using the SUM function to sum a list of variables takes CPU 2.04 seconds, while the DIY method with SAS operators
takes CPU 1.93 seconds. The ARRAY statement uses CPU 2.88 seconds to complete the task. There’s not a great
deal of performance improvement gained in the DIY method. If the function is performing more complex operations,
it’s certainly better to rely on the function because the code is simpler and easier to maintain.
The following SAS macro program helps demonstrate stored program usage. The macro produces SAS code with a
specified number of IF statements and with a different specified number of passes. This macro generates SAS code
that in turn generates other SAS code. The program is run as a single DATA step and as a stored program for
comparison.
filename sascode temp;
%macro runtest(dsname,npasses,nlines);
data _null_; file sascode;
put "do npasses=1 to &npasses;";
put " x=ceil(ranuni(13131)*&nlines);";
do i=1 to &nlines;
put @5 'if x=' i ' then y=' i ';';
end;
put @5 'end;';
put 'run;';
data &dsname.; %include sascode;
data &dsname./pgm=&dsname.; %include sascode;
data pgm=&dsname.; execute; run;
%mend;
6
SUGI 30 Applications Development
With 1000 passes, but only 10 statements, the compile-and-run DATA step is a little faster than the stored program.
The compilation time isn’t counted toward the final execution time. This example shows that the compile phase for a
small program is a tiny fraction of the overall execution, so creating a stored program is not very beneficial to overall
performance.
Here the compile-and-execute time is greater than when running only the stored program. In those rare cases where
the program size is indirectly proportional to execution passes, the stored program might be useful.
Be aware that if a stored program contains macros, they are resolved at compilation time and have no further
bearing. In this example, the &MYVALUE macro variable is reset to second, but it has no bearing when the stored
program is run a second time.
1 %let myvalue=first;
2 data xyz/pgm=xyz;
3 x="&myvalue.";
4 put x=;
5 run;
x=first
8
9 %let myvalue=second;
10 data pgm=xyz; run;
x=first
We can conclude that the larger the compiled code, with fewer execution passes, the more beneficial the stored
program can be.
7
SUGI 30 Applications Development
data cntlin;
fmtname='testfmt';
do start=1 to 5000;
label=put(start,z4.);
output;
end;
run;
This example illustrates how to use the format for key lookup. The PUT function uses the format to create a
character value. The variable key1x contains the label for the corresponding key1 value.
MERGING
The more traditional way to perform key lookup is a merge. To perform a merge, the input data set must first be
sorted by the key. The sorting step is included in the overall CPU time.
30 if want;
31 run;
Note that the CPU time using the PUT function was .54 seconds. The sort, along with the merge, takes .65 + .29
seconds, or .94 CPU seconds, as compared to .54 CPU seconds for the format usage. But you might need an
additional sort. Remember that if the necessary sort disrupted the observations so that they needed to be sorted
again, more CPU time is consumed as shown here:
The format and PUT function method does not disrupt the observation order. Multiple keys can be handled within the
same pass using the PUT function.
But for SORT/MERGE, you must sort and merge by each key separately, adding greatly to CPU time usage.
INDEXING
To illustrate using an indexed data set, the next step is to create the CNTLINX data set from the CNTLIN data set,
adding an index so that a key lookup can be performed to locate the corresponding label value.
9
SUGI 30 Applications Development
Looking for the same key two times in a row is a problem that requires the introduction of a dummy key. Adding the
dummy key adds to the execution time. The real time increases dramatically: 26.85 seconds as compared to less
than 1 second using the other approaches.
HASH OBJECT
The hash object is a new feature in SAS®9 and a foray into object-oriented programming.
Populating the hash table along with all the searching took only .82 seconds. Although a little longer than the other
methods, the hash object method is faster when you consider that the setup time is included. Notice that the hash
table does not persist across DATA steps and must be re-created each time it is used.
THREADS OR NOTHREADS
To demonstrate threading in PROC SUMMARY, the first step is to create a large data set with over 7 million
observations. There are about 500 different values of the CLASS variable and up to 30,000 observations per class.
The values of x are random, with y and z being 1 and 2 larger, respectively.
data temp;
do cv=1 to 500;
classvar=ceil(ranuni(13131)*500);
nobs = ceil(ranuni(13131)*30000);
do i=1 to nobs;
x = ceil(ranuni(13131)*1000);
y = x + 1;
z = x + 2;
output;
end;
end;
keep classvar x y z;
run;
NOTHREADS OPTION
23 proc summary data=temp nothreads;
24 class classvar;
25 format classvar z5.;
26 var x y z;
27 output out=new
28 mean=mx my mz min=minx miny minz
29 var=varx vary varz;
30 run;
10
SUGI 30 Applications Development
THREADS OPTION
31 proc summary data=temp threads;
32 class classvar;
33 format classvar z5.;
34 var x y z;
35 output out=new
36 mean=mx my mz min=minx miny minz
37 var=varx vary varz;
38 run;
The first example ran explicitly with the NOTHREADS option. CPU time and real time match at 16.31 seconds. Note
that we use a FORMAT statement, multiple variables, and multiple statistics to compute this. Run with the THREADS
option, the real time for the same example is less than the CPU time because multiple threads run simultaneously,
and the CPU time is the sum of the thread execution time. The real time, 13.93 seconds, is a reduction over the
NOTHREADS run, which is the desired result.
The following examples show how different hardware platforms can influence performance and how each platform
can have significantly different performance indicators. To set up the example, the number of CLASS variables is
increased and the examples are run with the NOTHREADS and THREADS options on two different UNIX machines.
Using the THREADS option on UNIX 1 is a considerable improvement in real time, going from 1:10.76 down to
32.36, using only 38.40 more seconds of CPU. While UNIX 2 has somewhat similar improvements in real time, the
THREADS option used more than twice the CPU time as the NOTHREADS option.
To benefit from threading, you need large amounts of data and you need to run on a multiprocessing machine. Not
all SAS procedures are threaded, although more procedures are being converted to support threading capability.
The THREADS/NOTHREADS global option can control whether any procedure will run with threading enabled.
11
SUGI 30 Applications Development
CONCLUSION
This paper provides examples of several choices that the SAS programmer can make regarding efficiency. In each
case, there are advantages and disadvantages to the choices. The SAS programmer should consider all aspects of
the application to make the appropriate choice.
RESOURCES
Ray, Robert. 2000. “Version 8 Base SAS® Performance: How Does It Stack Up?” Proceedings of the Twenty-fifth
Annual SAS Users Group International Conference. Available
http://www2.sas.com/proceedings/sugi25/25/aa/25p009.pdf.
Ray, Robert. 2003. “An Inside Look at Version 9 and 9.1 Threaded Base SAS® Procedures.” Proceedings of the
Twenty-eighth Annual SAS Users Group International Conference. Available
http://www2.sas.com/proceedings/sugi28/282-28.pdf.
Shamlin, David. 2004. “Threads Unraveled: A Parallel Processing Primer.” Proceedings of the Twenty-ninth Annual
SAS Users Group International Conference. Available http://www2.sas.com/proceedings/sugi29/217-29.pdf.
CONTACT INFORMATION
Your comments and questions are valued and encouraged. Contact the author:
Rick Langston
SAS Institute Inc.
SAS Campus Drive
Cary, NC 27513
Phone: (919) 677-8000
E-mail: [email protected]
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS
Institute Inc. in the USA and other countries. ® indicates USA registration.
Other brand and product names are trademarks of their respective companies.
12