Scribd Logo
Upload

Welcome to Scribd!

  • 44 views

    Parallel Processing in Sas: Pooja Matekar

    Uploaded by

    rajeshmsit
    This document describes how to introduce parallelism through code in SAS to significantly increase performance when working with large datasets. It provides a generalized code approach that uses macro variables to split the input dataset, generate multiple code instances, execute them in parallel batches, aggregate the output datasets, and concatenate the log files. Introducing parallelism through this method allows leveraging multiple CPUs without needing to modify the original code.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Parallel Processing in Sas: Pooja Matekar

    Uploaded by

    rajeshmsit
    44 views4 pages
    This document describes how to introduce parallelism through code in SAS to significantly increase performance when working with large datasets. It provides a generalized code approach that uses macro variables to split the input dataset, generate multiple code instances, execute them in parallel batches, aggregate the output datasets, and concatenate the log files. Introducing parallelism through this method allows leveraging multiple CPUs without needing to modify the original code.

    Original Description:

    Feroz Ticket

    Original Title

    Feroz Ticket

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document describes how to introduce parallelism through code in SAS to significantly increase performance when working with large datasets. It provides a generalized code approach that uses macro variables to split the input dataset, generate multiple code instances, execute them in parallel batches, aggregate the output datasets, and concatenate the log files. Introducing parallelism through this method allows leveraging multiple CPUs without needing to modify the original code.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    44 views4 pages

    Parallel Processing in Sas: Pooja Matekar

    Uploaded by

    rajeshmsit
    This document describes how to introduce parallelism through code in SAS to significantly increase performance when working with large datasets. It provides a generalized code approach that uses macro variables to split the input dataset, generate multiple code instances, execute them in parallel batches, aggregate the output datasets, and concatenate the log files. Introducing parallelism through this method allows leveraging multiple CPUs without needing to modify the original code.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 4

    COGNIZANT TECHNOLOGY SOLUTIONS

    PARALLEL PROCESSING in SAS


    Pooja Matekar
    12/9/2014
    Introduction:
    When we work with large data sets and large volumes of SAS processes, leveraging
    multiple CPUs to run parts of a SAS job in parallel threads can significantly increase the
    speed of performance. There are several ways to multi-thread jobs in SAS, for example, we
    can use threaded procedures in SAS 9 or run parallel jobs by using MP Connect in
    SAS/CONNECT 8 or SAS/CONNECT 9. However, in many scenarios, we cannot take
    advantage of these methods.

    Parallel processing or multi-threading a SAS job is a mode of execution in which two or


    more portions of a job run in parallel streams to accelerate processing and reduce the real
    time to completion. By using the technique that is explained here, we can significantly
    increase performance time. However, this result also is affected by the number of CPUs that
    are available.

    Let discuss one of the approach i.e. introducing parallelism through code

    Parallel processing through code:


    This technique is based on a mechanism that spawns multiple jobs from a single instance of
    code, and, depending on how your SAS code is structured; either the whole process or parts
    of the process can be run in parallel.

    Using this technique the user is required to execute just a single copy of code. Through this
    code many codes will be generated and executed. Once the code is submitted, splitting the
    input dataset, generation of multiple instances of the code, their execution and aggregation
    of output datasets all follow automatically. Because of automation there is no need to
    monitor spawned jobs.

    Following is the generalized version of the code, which can be used in your code to
    implement parallelism. %PARALLEL will include your code (this part is given in bold).

    PROC PRINTTO LOG="./redirected.log" NEW; RUN;

     Using PRINTTO procedure will help us to keep our main log clean. We will redirect the
    details of parallel is used to redirect the log to some other location with the specified name.
    %LET JOBNAME = MYJOB ;
    *** Job name must be a unique name and should NOT have extensions **;
     This job name will used to create multiple SAS scripts.
    %LET NUM_PARTITIONS = 10; * Number of threads;
     Num_Partitions will decide the number of threads i.e. the number of independent SAS
    scripts to be created.

    Parallel Processing Page 2


    %MACRO PARTITION;
     This is the macro where multiple scripts will be created.
    OPTIONS OBS=0 NOSYNTAXCHECK FULLSTIMER SOURCE2;

    %DO PART_NUM=1 %TO &NUM_PARTITIONS;


    FILENAME MPRINT "./&JOBNAME._PART&PART_NUM..sas" LRECL=170;
    OPTIONS MPRINT MFILE;
    .
    .
    YOUR CODE
    .
    .
    %END; *NUM_PARTITIONS;
     The above DO loop will create multiple SAS scripts using the MPRINT and MFILE macro
    options.

    OPTIONS OBS=MAX NOMFILE;


     As macro %partition iterates and generates SAS codes, it also executes the statements
    contained in this macro. If it run as it is, it would defeat the purpose of multi-threading, by
    creating datasets sequentially rather than concurrently .To avoid this we set the option
    OBS=0 which makes the program momentarily iterate through all partitions. Thus we
    practically do not spend any processing time.
     When the codes are generated we set OBS=MAX.

    *** Create a shell script ***;


    DATA _NULL_;
    FILE "RUN_&JOBNAME..sh";
    %DO PART_NUM = 1 %TO &NUM_PARTITIONS;
    PUT "sas &JOBNAME._PART&PART_NUM..sas -log &JOBNAME._PART&PART_NUM..log &";
    %IF %SYSFUNC(MOD(&PART_NUM, 20)) = 0 %THEN %DO;
    PUT "wait";
    %END; *SYSFUNC;
    %END; *PART_NUM;
    PUT "WAIT";
    RUN;
    %MEND PARTITION;

    %PARTITION;
     The above do loop will create a shell script for each of the SAS script. If loop will check
    whether all the scripts in that particular group have executed, if yes then it will proceed
    further with other set of scripts. If not then it will wait for all the scripts in that group to
    complete before it starts executing the rest scripts.

    Parallel Processing Page 3


     The numeric 20 over here specifies that the SAS scripts will execute in a batch of 20. Each
    batch will contain 20 scripts. First 20 scripts will execute in parallel. After these scripts are
    executed next set of 20 scripts will start executing this process continues till the last script
    is executed.

    *** Execute the shell script ***;


    X "chmod 755 RUN_&JOBNAME..sh";
    X "RUN_&JOBNAME..sh";
    X "wait";
     The above piece of code will execute the shell scripts created in previous step.

    %DO X=1 %TO &NUM_PARTITIONS .;


    X " cat &JOBNAME._PART&PART_NUM..log >>
    &outlog.&run_dt._&sysuserid._&sysmacroname._%upcase(&option.).log";
    %END;
     This do loop will concatenate the different log files to the main log.

    ***Concatenate all the datasets***;


    DATA OUT_LIB.OUTPUT;
    SET %DO PART_NUM = 1 %TO &NUM_PARTITIONS;
    LIB.OUTPUT_&JOBNAME._PART&PART_NUM
    %END;;
    RUN;
     The above data step will aggregate all the intermediate datasets. This dataset will be our
    final dataset.

    PROC PRINTTO; RUN;


     Proc printo without any options will redirect the log to main log.

    ***Optional Part: Delete intermediate scripts and logs***;


    %DO X=1 %TO &NUM_PARTITIONS .;
    X " rm &JOBNAME._PART&PART_NUM..sas ";
    X " rm &JOBNAME._PART&PART_NUM..log ";
    %END; *PART_NUM;
    X " rm &JOBNAME..sh";
     The final DO loop will delete the intermediate scripts and log files. And lastly the shell script
    will be deleted.

    Parallel Processing Page 4

    You might also like