INTLDM - USER DOCUMENTATION ============================ INTRODUCTION INTLDM is a program for integrating spot intensities from Laue X-ray diffraction images. It makes use of the data in a Laue Data Module (LDM) parameters file. This file must contain refined setting parameters which give a very close match between observed and predicted spot positions. An expanded LDM parameter set is used. This is also used by LAUEGEN and contains some integration related parameters in addition to the standard LDM parmeters. The program makes use of a set of LDM compatible integration routines written by Hao Quan and the program itself was written by John Campbell (CCLRC Daresbury Laboratory). List of sections: The Extended LDM Parameter Set Running the Program Interactively Running the program in background or batch mode Note on Command Line Parameters The Log File Output The Output Intensities File Program Function Bad Spots Diagnostics File Format Control Data Examples References THE EXTENDED LDM PARAMETER SET In addition to the standard LDM parameter set, the following additional parameters have been defined for use in INTLDM and the corresponding integration option in LAUEGEN. 1) INT_FTYPE This indicates the type of output file required for the integrated intensities. It may either be 'mtz' (the default) to output an MTZ relection file or 'ge' to output the old style .ge1/.ge2 files. 2) INT_TEMPLATE This provides a template from which the output integrated intensities file names are constructed. It must contain a field of hash characters which will be replaced by the pack identifier number when the output file name is formed. The default template is 'int####.mtz'. If the output file type is 'ge' then two output file names will be constructed with extensions .ge1 and .ge2; any given extension will be ignored in this case. If the output file type is 'mtz' then the file extension will be as specified in the template. 3) NPROF This is the number of bins to be used for profile determination and fitting. These are selected radially around the centre of the pattern. The default value is that returned by the routine LFN_SAINT_DF and, at the time of writing, is 8. The value must be an integer in the range 1-17. 4) PROF_ROTATE This is a flag indicating whether or not to rotate profiles data to theta = 0.0 radially around the beam direction when forming profiles from elliptical spots ('yes' or 'no'). The default value is 'yes'. If profile rotation is used then a reverse rotation is applied to the profile when a spot is integrated. The potential advantage of taking account of the exact radial angles and possibly variable lengths of the spots must be offset against a degree of inaccuracy resulting from the pixel mapping process involved. No rotation is done when spots are circular. 5) PROMIN This is the minimum intensity for which spots are to be included in the determination of profiles. The default value is 0.0 which indicates an undefined value. If the default value is used then INTLDM will use a value of 600.0 for image-plate data and a value of 100.0 for film data. Any positive real value is valid. 6) PROF_ALL This flag indicates whether profiles are to be determined for each plate of a multi-plate pack (code = 'yes') or whether they are only to be determined using the key (normally the first) plate of the pack (code = 'no'). The default is 'no'. The keyword has no effect if there is only a single image in a pack. 7) OVLIM This is the number of overloaded pixels allowed within a spot peak for the integrated intensity still to be considered as good. If there are more than this number of overloaded pixels, the intensity will be flagged as an intensity overload. 8) SCALE_INT This is a scale factor to apply to the output integrated intensities before being written to the output file. This is important if the file output type is 'ge' as the output intensities are stored in this case as integers up to 32767. The default value is 0.0 to indicate an undefined value. If the default value is used then INTLDM will use a value of 0.1 for image-plate data and a value of 1.0 for film data.Any positive real value is valid. RUNNING THE PROGRAM INTERACTIVELY The program may be run in an interactive mode by typing the command: laue intldm Use the default 'terminal' as the reply to the 'DATA:' prompt. The following prompt is then output: DIAGNOSTIC SPOTS FILE (optional): A file name is given only if additional diagnostics are to be output to the program log file for the integration of individual spots specified in a diagnostics spot list file. If specified, the named file must exist; its format is described below. The program then runs and a question and answer sequence is entered as follows: Log file name (default ext=.log): The reply is the name of the output log file if one is required. If no file name is given, no log file will be written. If the file already exists then an additional quetion is asked: Overwrite (y/n) [y]: The reply is 'y', the default, to overwrite the file or 'n' if it is not to be overwritten. In the latter case, the 'Log file name' prompt will be repeated. LDM parameters file name (default ext=.ldm): The reply is the name of the LDM parameters file. This may contain the extended LDM parameters for the integration in addition to the standard LDM parameters. The next six prompts allow for the current values of the integration specific extended LDM parameters to be overridden. The default values in the prompts will be the values read from the LDM file. In the description below, the default values are shown. Output file type (mtz/ge) [mtz]: The reply is 'mtz' or 'ge' depending on the required type of file for the output integrated intensities. Template for intensity file names [int####.mtz]: The reply is the template string from which the output file names are to be formed. Minimum intensity for profiles [600.0]: The reply is the minimum intensity value for which spots are to be included in the formation of spot profiles. Number of bins for collecting profiles [8]: The reply is the number of radial bins from 1 to 17 to be used for gathering spot profiles. Rotate profiles for elliptical spots [y/n] [n]: The reply is 'yes' to rotate spots when forming profiles for elliptical spots or 'no' if this option is not required. Form profiles for each plate in pack (y/n) [n]: This question will only be asked if there is more than one plate per pack defined in the LDM file. The reply is 'yes' if profiles are to be determined separately for each plate in the pack or 'no' if profiles are only to be determined for the key plate in the pack. Number of overload pixels allowed in a spot [0]: This is the number of overloaded pixels allowed within a spot peak for the intensity still be be treated as good (non-overloaded). Scale for output intensities {0.1}: The reply is the scale factor to be applied to the integrated intensities before writing them to the output file. Number of pack to integrate (0=all): The reply is the number of the pack to be integrated or '0' to integrate all packs defined in the LDM file. Error messages will be output if invalid replies are given and the corresponding prompts will be repeated. RUNNING THE PROGRAM IN BACKGROUND OR BATCH MODE The program may be run in background or batch mode by typing the command: laue intldm The following prompt/reply sequence is entered: DATA [terminal]: The reply is the name of the required LDM parameters file. This should contain any non-default values required for the extended set of LDM parameters relating to the integration as well as the standard LDM parameters. Alternatively a control file may be prepared and input in which only the integration specific parameters are given together with an indirect reference to the required LDM file. PACK NUMBER (0=all) [0]: The reply is the number of the pack to be integrated or '0' to integrate all packs defined in the LDM file. DIAGNOSTIC SPOTS FILE (optional): A file name is given only if additional diagnostics are to be output to the program log file for the integration of individual spots specified in a diagnostics spot list file. If specified, the named file must exist; its format is described below. Background or batch mode and the log file name are input in response to further standard prompts from the 'laue' command procedure. NOTE ON COMMAND LINE PARAMETERS If the program is not being run using the 'laue' command then it is necessary to know that the program can have command line arguments. These are either a pack number or the code 'all' and '-d' or 'd' followed by the name of a diagnostic spots file. The pack specification is only necessary when running the program in a background or batch mode; in interactive mode the program will prompt for it. The diagnostic spots file specification is optional. If the parameter '-b' or 'b' is given, then the program will read its input as if in background/batch mode without prompts even if it is not run in background or batch mode. When running the program the environment variable (or logical name under VMS) SYMOP must point to the symmetry operators file e.g. symop.lib. THE LOG FILE OUTPUT After an initial heading giving the program name and run date and time, the values of the (extended) LDM parameter set are listed. In interactive mode a note is then made of any of the integration parameters which have been altered via the prompt/reply sequence. The specific integration parameters are then listed with descriptive strings. The log file records the packs and plates for which integrations and/or spot profile determinations have been carried out. A character plot is output for the spot profile determined for each of the spot profile bins. An example is as follows: Profile 1 (0.0-30.0) ................. ................. ................. ......00000...... ....000000000.... ...000*****000... ..00**01110**00.. ..00*0146520*00.. .00*0019FC400*00. ..00*017EB41*00.. ..00**02441**00.. ...000*****000... ....000000000.... ......00000...... ................. ................. ................. The profile number is followed by the angular range in degrees of the profile bin. The background subtracted profile values are scaled to a range from 0 to 15 represented as hexadecimal digits from 0-9 and A-F. A border of '*' characters 1 pixel thick shows the boundary between the spot and background regions. Numbers outside this border are background subtracted background region values and should ideally be zero. Points within the integration box which are not part of the spot, border or background area are represented by '.' characters. If diagnostic information was requested for selected spots, then the following information and plots are output for each spot as exemplified below: Integrated spot -2 12 1 at x = 425.7, y = 664.0 rasters Intensity = 24589.9, sig(I) = 2048.6 Background plane parameters: a = -0.515E-01, b = 0.808E+00, c = 0.134E+03 Number of rejected pixels = 14 Pixel values plot scaled by 0.266E-01 to give a maximum value of 99 . . 4 3 3 4 4 4 4 . . . . . 4 3 4 4 4 4 4 4 4 . . . 5 4 4 3 4 5 6 5 4 4 3 3 . 4 4 4 4 7 14 16 8 5 4 3 4 . 4 4 4 5 19 49 50 24 6 4 4 3 4 4 3 4 6 34 93 99 44 10 4 4 4 4 3 3 3 5 19 76 77 23 7 4 4 4 4 . 4 4 4 8 23 17 6 4 4 4 4 3 . 4 4 3 4 4 4 4 4 4 4 4 4 . . . 4 4 3 3 3 3 3 3 4 . . . . . 3 3 3 3 3 4 3 . . Background subtracted pixel values plot scaled by 0.266E-01 . . 0 R R 0 0 0 0 . . . . . 0 R 0 0 0 0 0 R 0 . . . R R 0 * * * * * R 0 R 0 . R R * 0 3 10 13 4 * * 0 0 . 0 0 * 2 16 45 46 21 2 * * 0 0 0 R * 2 31 90 95 40 6 1 * 0 0 0 0 * * 15 73 73 19 3 1 * 0 0 . 0 0 * * 19 13 3 1 1 * 0 0 . 0 0 0 R * * * * * 0 0 0 . . . 0 0 R 0 0 0 0 0 0 . . . . . 0 0 0 0 0 0 R . . Spot profile plot scaled to a maximum of 99 . . 0 0 0 0 0 0 0 . . . . . 0 0 0 0 0 1 0 0 0 . . . 1 0 0 * * * * * 0 0 0 0 . 0 0 * 0 2 16 13 8 * * 0 0 . 0 0 * 3 20 85 54 54 6 * * 0 0 0 0 * 3 29 85 99 46 7 1 * 0 0 0 0 * * 10 65 65 46 6 1 * 0 0 . 0 0 * * 18 13 4 0 0 * 0 0 . 0 0 0 0 * * * * * 0 0 0 . . . 0 0 0 0 0 0 0 0 0 . . . . . 0 0 0 0 0 0 0 . . The spot indices, spot position on the image in raster units, the spot intensity and its standard deviation are listed. This is followed by the parmeters describing the background plane determined for the spot and the number of rejected backgound pixels. Three plots follow: 1) Pixel values plot. This plot lists the pixel values associated with the given spot scaled to a range of 0 to 99. The scale factor used is noted. The integration box shows the values of the pixels within the areas associated with the spot, border and background regions. Pixels outside these areas are represented by '.' characters. 2) Background subtracted pixel values plot. The integration box shows the background subtracted values of the pixels within the areas associated with the spot and background regions. The pixels are on the same scale as for the first plot. Pixels which have been rejected from the background calculation are represented by the character 'R'. The border pixels are shown as '*' characters and separate the spot from the background region. Pixels outside these areas are represented by '.' characters. For spatially overlapped spots, overlapped pixels which have been excluded because they are overlapped by neighbouring spots are marked by the character 'X'. Overloaded pixels which have been excluded from the spot peak are marked by the character '+' (black). 3) Spot profile plot. This plot shows the profile used when determining the profile fitted intensity for the spot. The profile pixels are scaled to a range from 0 to 99. The border pixels are shown as '*' characters and separate the spot from the background region. Pixels outside the spot, border and background areas are again represented by '.' characters. For spatially overlapped spots, overlapped pixels which have been excluded because they are overlapped by neighbouring spots are marked by the character 'X'. Overloaded pixels which have been excluded from the spot peak are marked by the character '+' (black). At the end of each integration, a table of statistics is output with a key being given only after the first image integrated. An example is given below: Table of Intensity Statistics ----------------------------- Spot type Total Good Ovld. Neg. Bad Mean Mean Number Number --------- Spots Spots Spots Spots Spots I I/sigI >3sigI >10sigI All Spots 4409 3606 1 261 0 4888.6 6.7 2709 552 Singles (sep) 3160 3154 0 259 0 3005.7 6.4 2261 423 Multiples (sep) 454 452 1 2 0 16925.7 8.9 448 129 SpOv. Singles 746 0 0 0 0 0.0 0.0 0 0 SpOv. Multiples 49 0 0 0 0 0.0 0.0 0 0 Key: Good = Good spot, measured and not bad/overload (sep) = Spatially separated Ovld. = Intensity overload SpOv. = Spatially overlapped Spots are divided into five categories: * All Spots. * Singles (spatially separated). * Multiples (spatially separated). * Singles (spatially overlapped). * Multiples (spatially overlapped). For each category, the following items of information are output: * The total no. of spots predicted. * The number of good spots; i.e. integrated and not bad/overload. * The number of spots with overload pixels present. * The number of spots with negative intensities. * The mean spots intensity (non-negative intensities). * The mean Intensity/sigma(Intensity) value (non-negative intensities). * The number of spots with intensities > 3.0*sigma(Intensity). * The number of spots with intensities > 10.0*sigma(Intensity). Notes: 1) There may be a few predicted spots for which the integration box used does not lie fully within the image. These are not integrated. 2) Spots with negative intensities are also included in the good spots category. 3) Spots with no more than OVLIM overloaded pixels are also included in the good spots category. 4) For the counts of spots with intensities > 3 sigma and > 10 sigma, good spots and intensity overloads are included. THE OUTPUT INTENSITIES FILE Two possible output file formats are available. These are the .ge1/.ge2 files as originally used in the Daresbury Laboratory Laue Software Suite and a newer MTZ format as specified in 1995. The MTZ file has the following columns: H Measured 'h' index K Measured 'k' index L Measured 'l' index PACK_ID Pack identifier PLATE Plate number within pack XF 'x' coordinate of spot in mm from pattern centre (ideal) YF 'y' coordinate of spot in mm from pattern centre (ideal) LAMBDA Wavelength (Angstroms) I Integrated intensity (real) SIGI Sig(I) (0.0 = not integrated) MULT Multiplicity MINHARM Minimum harmonic number MAXHARM Maximum harmonic number NOVPIX The number of overloaded pixels found in the spot when being integrated (-1 if not determined) FLAGS bit 0 spatial overlap bit 1 too close to integrate bit 2 overload bit 3 bad spot (other than overload) PROGRAM FUNCTION INTLDM is a program for integrating spot intensities from Laue X-ray diffraction images. It makes use of the data in a Laue Data Module (LDM) parameters file (Campbell, Clifton, Harding & Hao, 1995). This file must contain refined setting parameters which give a very close match between observed and predicted spot positions. This refinement will normally be done using the LDM based version of the program LAUEGEN (Campbell, 1995). An expanded LDM parameter set is used. This is also used by LAUEGEN and contains some integration related parameters in addition to the standard LDM parmeters. The program makes use of a set of LDM compatible integration routines (SAINT - Stand Alone INTegration) written by Hao Quan and based on the method of Rossmann (1979). The program itself was written by John Campbell (CCLRC Daresbury Laboratory). The integration procedure involves the following: 1) Profile shape. An image is divided into 1 to 17 radial bins. All profile shapes use elliptical masking as standard and treat a circular spot as a special case of an ellipse when both axes are equal. If profile rotation is requested, then account is taken of the lengths and theta angles around the beam direction of the individual spots used to form the profiles. The spots forming the profile are rotated to an angle theta=0.0 radially around the beam centre position. When using the profile for integrating a spot, the reverse rotation is applied to the bin profile. These rotations involve the mapping of nearest pixel points and the advantage of taking account of the ellipse orientations and possibly variable lengths must be offset against any loss of accuracy due to the mapping process. Profile rotation may well be most beneficial if the spots cover many pixels but should probably be used with caution if the spots only cover a few pixels. Profile rotation is never done for circlular spots i.e. those defined with the SPOT_WIDTH parameter = 0.0. 2) Background determination. The three background plane constants: a, b, c are given by setting Sigma(rho - ap - bq - c)**2 n to a minimum over 'n' background points in a spot box, where 'rho' is the optical density, 'p' and 'q' are the two-dimensional positions of the pixel. The background plane is re-calculated by omitting pixels outside the expected Gaussian distribution about mean background. 3) Forming profiles. 'N' spots which satisfy following conditions are selected to form a profile in every bin of the image: * Non-overlapped. * Integrated intensity 'J' greater than the requested PROMIN value Minimising (JP - rho + ap + bq + c) gives profile 'P' for every point within the spot box: 4) Profile fitting. The integrated intensity by profile fitting is given by: I = Sigma P(rho - ap - bq - c) * Sigma P / Sigma P**2 m m m 5) Standard deviation. The integrated error, 'sigma', in fitting the profile can be determined from the differences between measured optical densities and the scaled profile. Thus, sigma**2 = sigma**2(P) + m / n*sigma**2(B) where sigma(P) and sigma(B) are the standard errors in determining the peak and background regions within the reflection box, respectively: sigma**2(P) = Sigma [JP - (rho - ap - bq - c)]**2 m sigma**2(B) = Sigma rho**2} - 1 / n*Sigma rho**2 n n 6) Spatially Overlapped Spots The program will integrate spots which are spatially overlapped provided they are not closer than the distance set for the LDM parameter SPOT_EPSILON. If the deconvolution of spatially overlapped spots is not required, then a large value should be given for SPOT_EPSILON. Spots are only marked as 'too close to integrate' if they have first been flagged as spatially overlapped i.e. the SPOT_EPSILON value only affects the integration criterion and not the overlap criterion. For spatially overlapped spots, any pixels which belong to the peak areas of neigbouring spots are excluded from the set of pixels used in background calculation and from the set of pixels used in determining the profile fitted intensities. 7) Spots with Overloaded Pixels Using profile fitting, spots with only a few overloaded pixels can still be integrated to give good intensities. These intensities are determined by omitting the pixel positions of the oveloaded pixels from profile fitting mask when a spot is integrated. The number of overloaded pixels allowed within a spot, to be treated in this way, is set by the user (the OVLIM extended LDM parameter) and an appropriate number will clearly depend on the number of pixels in the spot. The intensity derived from any spot with more than the requested limit of overloaded pixels is flagged as being an intensity overload. BAD SPOTS An integrated spot is flagged as bad and a bit is set in the integration code flag if any of the following occur: * (bit 0) No non-overlapped/non-overloaded peak pixels . * (bit 1) Less than 20% of peak pixels are non-overlapped/non-overloaded. * (bit 2) Less than three non-overlapped background pixels. * (bit 4) Less than 20% of background pixels are non-overlapped. The 20% (approximate) values are calculated from the expression (NPIX-3)/5 where NPIX is the number of pixels is the peak or background as relevant. DIAGNOSTICS FILE FORMAT The diagnostics file consists of a record for each reflection to be examined with five integer numbers being given as follows: ipack iplate h k l where 'ipack' is the pack number, 'iplate' is the plate number and 'h', 'k' and 'l' are the reflection indices. CONTROL DATA EXAMPLES 1) LDM data file with extended LDM parameters included:- SYSTEM Tetragonal LATTICE P SYMMETRY 96 BEAM_AXIS +b* ROTATION_AXIS +a* IMAGE_TYPE ip IMAGE_DATA i2 NXRAST 1200 NYRAST 1200 RASTER_SIZE 150.0 RMIN 10.0 RMAX 90.0 DISTORTION_TYPE r/toff FTHRESH 125.0 FIDBOX 5.0 STHRESH 200.0 OVERLOAD_PIXEL 65000 OVLIM 2 NUMPACK 1 NPLATES 1 TEMPLATE img####.od SPOT_EPSILON 0.05 SPINDLE 0.0 PACK_ID 1 FILENAME[1][1] /priv1/jwc/lys.i2 X_CEN_F 583.0 Y_CEN_F 593.0 A 79.19 B 79.19 C 37.968 ALPHA 90.0 BETA 90.0 GAMMA 90.0 LMIN 0.4 LMAX 2.2 DMIN 2.4 PHIX 94.342 PHIY -15.516 PHIZ -103.653 CTOF 199.826 Y_SCALE 1.00010 SPOT_LENGTH 0.8 SPOT_WIDTH 0.75 SPOT_FACTOR 0.0 SPOT_BORDER 2.0 SPOT_DELTA 0.8 X_C 0.2002 Y_C -0.5628 W_C 0.0 TWIST -4.640 TILT -22.491 ROFF 10.309 TOFF 25.966 RFL 1 INT_FTYPE mtz INT_TEMPLATE lys###.mtz NPROF 12 PROF_ROTATE yes PROMIN 600.0 PROF_ALL no SCALE_INT .1 2) Control file with an indirect reference to a standard LDM file:- @lys.ldm INT_FTYPE mtz INT_TEMPLATE lys###.mtz NPROF 12 PROF_ROTATE yes PROMIN 600.0 PROF_ALL no SCALE_INT .1 REFERENCES 1) Campbell, J.W. "LAUEGEN, an X-windows-based program for the processing of Laue X-ray diffraction data" J. Appl. Cryst. (1995) 28pp. 228-236 2) Campbell, J.W., Clifton, I.J., Harding, M.M. & Hao, Q. "A Laue Dat;a Module (LDM) for use in the processing of Laue X-ray diffraction data" J. Appl. Cryst. (1995) 28pp. 635-640 3) Rossmann M. Acta. Cryst. (1979) 12pp. 225-238