Summary

• which files to sort (if not standard input)
• what kind of files: fixed length (binary), or text file
• what are the sort keys
• where to put the final result (if not standard output)
• what resources are to be used: memory, directories for temporary sortworks, ...
• optional features (deduplicate, filtering, ...)

2) At the OS command prompt --- a UNIX terminal or Windows cmd.exe ---, just type:

hxsm your-parmfile

3) Verbose switch:

hxsm -v your-parmfile

4) Check switch:

hxsm -v -c your-parmfile

Note that there is a command line with traditional switches described below, which may replace the parameter file in most cases. It is useful to specify input and output file names that come from environment variables.

Next section is the Parameter Statements.

For an easy start, see some Sample Sort Jobs below.

 OPTION  option,option,...  
    or 
 OPTIONS  option,option,...  

where options are:

 OPTION RSEP=|
  or
 OPTION RSEP=7C

 OPTION PROCS=8

 OUTFIL FILE=n,INCLUDE=(condition1[,AND/OR,condition2...]) 
   or
 OUTFIL FILE=n,INCLUDE=ALL

Example of OPTION statement usage:

 OPTION PROCS=4,KEEP_ORDER  ; force multithreading on 4 procs, force stable sort

 SORT or MERGE  FIELDS=(start,len,type,direction[,start,len,type,direction,..])
                    or
            FIELDS=(start,len,direction[,start,len,direction,..]),FORMAT=type
                    or
            FIELDS=ALL
                    or
            VFIELDS=(start,len,type,dir.[,start,len,type,dir.,..]),FIELDSEP=car

       start     : start of sort field (1,2, ... n):
                     position if fixed field (FIELD=)
                     field number if variable fields (VFIELDS=)

       len       : sort field length (FIELDS)
                   sort field max length (VFIELDS)

       type      : B or BI (Binary)      - binary field
                   C or CH (Char)        - normal ASCII characters (default)
                   I (Ignore case)
                   N or NU (Numeric)     - only '0' .. '9' characters (VFIELDS only)
                   P or PD (Packed)      - Packed decimal field 
                                           (Binary files only, last half byte = sign)
                   Y or Y2K (yy)         - 2 bytes numeric field containing year of a date
                                           (would contain '84' for year 1984)

       direction : A (Ascending order), D (Descending order)

       FORMAT    :  short form to use only when all sort keys have the same type

       FIELDSEP  : The FIELDSEP sub-parameter value can be a symbolic name, or a single
                   character, or an hexadecimal value.

                   Using symbolic name is recommended for general punctuation chars to 
                   avoid syntax parsing errors, specially with comments markers ';', '#'.

                   Symbolic names accepted:
                         BAR         =  the '|'  char
                         TAB         =  the Tabulation (X'09') char
                         COMMA       =  the ',' char
                         COLUMN      =  the ':' char
                         DIARESIS    =  the '#' char
                         SLASH       =  the '/' char
                         BACKSLASH   =  the '\' char
                         SEMICOLUMN   or SEMI-COLUMN   = the ';' char
                         SINGLEQUOTE  or SINGLE-QUOTE  = the "'" char
                         DOUBLEQUOTE  or DOUBLE-QUOTE  = the '"' char

                   Hexadecimal value: X'hh'  
                         (Hex syntax : UPPERCASE X, Singlequote, 2 Hex digits, Singlequote)

                   Default value is TAB

                   Examples:
                         VFIELDS=(.....),FIELDSEP=SEMI-COLUMN
                         VFIELDS=(.....),FIELDSEP=@
                         VFIELDS=(.....),FIELDSEP=        ; Field separator is space char
                         VFIELDS=(.....),FIELDSEP=X'7C'
                         VFIELDS=(.....)                  ; Default Field separator is TAB

MERGE verb is used to merge at least two files already sorted files.

To understand difference between SORT and MERGE, see SORT and MERGE discussion

One of SORT or MERGE statement is mandatory, unless using OPTION COPY

The FIELDS parameter describes sort keys at fixed column in the line (RECFM=V) or the record (RECFM=F).

FIELDS=ALL : the sort key is the whole record.

The VFIELDS parameter describes variable length sort keys separated by a given char.
This parameter cannot be used for Binary Files (RECFM=F).
When using VFIELDS, specify field separator with FIELDSEP=

Examples:

# these 3 SORT statements are equivalent:
 SORT FIELDS=(17,3,B,D,1,15,B,A)
     or
 SORT FIELDS=(17,3,BI,D,1,15,BI,A)
     or
 SORT FIELDS=(17,3,D,1,15,A),FORMAT=BI

 SORT FIELDS=(2,4,B,A) ; will sort a long (32 bits) in Descending order
 SORT FIELDS=(6,2,B,D) ; will sort a short (16 bits) in Ascending order

 SORT FIELDS=(12,20,I,A)

 SORT FIELDS=(7,4,P,D)  ;  7 BCD digits, with sign

 SORT VFIELDS=(12,20,I,A,7,9,N,D),FIELDSEP=:

 SORT VFIELDS=(12,20,I,A,7,9,N,D),FIELDSEP=

 SORT VFIELDS=(12,20,I,A,7,9,N,D),FIELDSEP=TAB
    or
 SORT VFIELDS=(12,20,I,A,7,9,N,D)  ;   FIELDSEP=TAB implied (default)

 SORT FIELDS=(25,2,Y,D,21,2,B,D,23,2,B,D)

 SORT FIELDS=ALL

  RECORD RECFM=record_format,LRECL=record_length

   record_format : F (Fixed), V (Variable) | T (Text), M ("MFCOBOL" or "MFVariable")
                              V and T are synonyms (equivalent)

   record_length : exact record length for RECFM=F, including any separator CR/LF
                   max record length for RECFM=V, excluding line separator CR/LF
 

The RECORD statement tells XSM if record type is fixed length or variable length.

The RECORD statement is mandatory.

With RECFM=V, LRECL is internally added by 2 to include CR/LF (Windows/UNIX flat text files)

RECFM=M is used for MicroFocus COBOL special variable format known as "MFCOBOL" or "MFVariable"


Examples:

 RECORD RECFM=F,LRECL=400

means that all the records have the same (fixed) length of 400 bytes

 RECORD RECFM=V,LRECL=133

means that this is a text file, and that the maximum line length is 133 (excluding CR/LF);

form 1:
 INPFIL filename         

      filename   : input file name

form 2:
 INPFIL DD:varname 

      varname   : Environment variable name holding input filename.  

The INPFIL statement describes input file names.

The INPFIL statement is optional: if omitted, the standard input (stdin) will be used as input file (UNIX style redirection and pipe are allowed).

one INPFIL line per input file

As of Versions 450/510, files may be specified 'a la MVS' by a ddname: DD:SORTIN
In that case, XSM will get the 'dsname' (file name) via the corresponding environment variable.

Examples:

 INPFIL C:\Myjob\BIGF.INP             # Windows
 INPFIL D:\TMP\Wrk.Dat                # Windows

 INPFIL /home/hh/bigf.inp             # UNIX
 INPFIL /home/hh/littlef.inp          # UNIX

 INPFIL DD:SORTIN1                    # any systems
 INPFIL DD:SORTIN2                    # any systems
 INPFIL DD:JOHNNY                     # any systems
# = actual file names via the environment variables SORTIN1, SORTIN2, and JOHNNY

The OUTFIL statement is optional: if omitted, the standard output will be taken as output file (redirection and pipe allowed).

Note: the output file may overwrite one of the input files, but in SORT operations only.

form 1 :

 OUTFIL filename

      filename   : output file name

form 2 :

 OUTFIL DD:varname

      varname    : Environment variable name holding output filename.  

form 3 :

 OUTFIL FILE=n,INCLUDE/EXCLUDE=(condition1,[AND/OR,condition2...)

      n          : number of the SORTOFn environment variable SORTOFn which holds 
                   the full path of output file

      condition  : start,length,datatype,operator,value
                start     : start pos of zone to compare
                length    : length of zone to compare
                type      : 'CH' (Char) or 'BI' (Binary)
                operator  : EQ or NE or LT or LE or GT or GE
                value     : C'xxxx' where xxxx = string to compare
                            Length of string must equal length of zone to compare
 

Examples:

 OUTFIL D:\TMP\BIGF.OUT    ;  OS2, WIN32 full pathname style

 OUTFIL /home/hh/bigf.out  ;  UNIX full path name style

 OUTFIL DD:FOO          
; all systems, actual file name via the environment variable FOO

 OUTFIL FILE=1,INCLUDE=(11,3,CH,EQ,C'MAR',OR,11,3,CH,EQ,'GAS')  
; output filename via environment variable SORTOF1

 OUTFIL FILE=2,OMIT=(11,3,CH,EQ,C'POP')                         
; output filename via environment variable SORTOF2

 SORTWORKS directory[,directory,...]

      directory : one or more directory full path for temporary work files.

The SORTWORKS statement is used to control where to put work files on different drives, or UNIX file-systems (recommended).

Examples:

 SORTWORKS C:\TMP,D:\TEMP   (OS2, WIN32)
 SORTWORKS /var/tmp,/tmp    (UNIX)

The SORTWORKS statement is optional: if omitted, all work files are created (then deleted after use) in the current directory.

*** Using SORTWORKS properly is essential for performance ***

When more than one sortworks directory is given, XSM will force multi-threading: one thread per directory.

When OPTION PROCS= is specified together with SORTWORKS=dir1,dir2,...,dir_n, XSM will use max of both values for threads number.

Using temporary workfiles on another physical disk than the one having input/output files will dramatically improve XSM performance as it will reduce concurrent I/O on a single disk.

Indeed, to simplify, consider XSM I/O phases are:

       Phase 1 :   reading input file  +  writing sortworks  

       Phase 2 :   reading sortworks   +  writing output file 

                                          (Disk1)    (Disk2)
   Phase 1 :   reading input file          Read
             + writing sortworks                      Write

   Phase 2 :   reading sortworks                      Read
             + writing output file         Write

                                          (Disk1)    (Disk2)
   Phase 1 :   reading input file          Read
             + writing sortworks           Write <==  Concurrent I/O on same disk: BAD!

   Phase 2 :   reading sortworks                      Read
             + writing output file         Write

This illustrates the same sort operation, with one disk / with two disks:

Please note than results highly depend on factors such as filesize, free memory, disk speed, CPU speed. Above chart just illustrates that using different physical hard disks will improve sort time with no doubt.

Temporary files are named after the current XSM Process Id, srtw04d2.xxx for instance for Pid 1234 (hexa 04d2). So running several XSM simultaneously using same SORTWORKS is not a problem.

Temporary files are deleted upon job completion.
In case of a severe error they may remain on disks so it is wise to check XSM return code.

 IOERROR IGNORE

The IOERROR statement is valid for variable length text (RECFM=V) files only.

It tells XSM what to do when a record is shorter than a defined key:

• if omitted, the first line shorter than the sort key will cause XSM to 'ABEND' (Abnormal end) immediately.
• if present, lines shorter than the sort key will be ignored.

 STORAGE size{K|M|G}

      size      : amount of main storage to use, in Kilobytes, Megabytes, or Gigabytes.

     Examples:

            STORAGE 450K
            STORAGE 2150K
            STORAGE 8M

The STORAGE statement is optional: if omitted, storage will be allocated 'au mieux' by XSM, depending on your Operating System and input files.

Nevertheless, it can be used to limit paging/swapping leading to System Stress in case of huge files:

        T = total input size in Megabytes
        T ≤ 64517 MB  :  STORAGE = square root( T ) / 4
        T > 64517 MB  :  STORAGE = T / 1020  
 

For instance, 3 input files of 25000 MB

            NONE                              ; do not suppress anything (default)
 OMIT       DUPKEYS (or DUPLICATE KEYS)       ; suppress records with duplicate keys
            DUPRECORDS (or DUPLICATE RECORDS) ; suppress duplicate records

form 2 : Filtering

 INCLUDE
 EXCLUDE     COND=(col,len,pattern[,startcol,len,pattern ... ])
 OMIT        COND=(col,len,type,op,pattern[,AND|OR,col,len,type,op,pattern ...])

      col        : start position of the field to be examined in each record (1..n)
      len        : length of the field
      type       : always 'CH' (useless but for IBM SORT compatibility)
      operator   : one of 'EQ' 'NE' 'GT' GE' 'LT' 'LE'
      pattern    : C'...'  string constant to be matched against the field

You cannot use EXCLUDE/INCLUDE operations with VFIELDS.

EXCLUDE and OMIT are synonyms.

Wild chars '*' and '+' MUST be escaped by a backslash '\' :

If the pattern does not contain the wild char '\*', its length should match the field length; otherwise, it will be truncated or padded with blanks to match the field length.

Successive triplets <start,len,pattern> specified in the same statement are processed with an 'OR' boolean operator

Successive statements (one statement per line) are processed with a 'AND' boolean operator.

Example:

 INCLUDE COND=(15,5,CH,EQ,C'JONES',OR,15,5,CH,EQ,C'SMITH')
 OMIT    COND=(11,3,CH,EQ,C'000')

These 2 filter statements mean:

 SKIP_HEADER nnn

The SKIP_HEADER statement will skip nnn first records (lines) from input file.

If more than one input file is specified, record are skipped from first file.

Example:

 SKIP_HEADER 4  # throw away report header