LSCALE - WAVELENGTH NORMALISATION AND ABSORPTION CORRECTION =========================================================== INTRODUCTION The program LSCALE is a program for the scaling and normalisation of raw integrated Laue intensity data to yield fully corrected structure amplitudes. The wavelength normalisation curve is normally derived internally from the Laue data using symmetry equivalent data recorded at different wavelengths. As an alternative, where appropriate, the normalisation curve may be determined by scaling the Laue data to a reference e.g. monochromatic set of data. The program also enables a wavelength and position dependent absorption correction to be calculated. Harmonic multiples data may be deconvoluted again making use of symmetry equivalent data recorded at different wavelengths. The program provides flexibility in the control of the refinement procedure. The refined scaling parameters are saved for re-use as required. The scaled data may be output in a number of formats as unmerged or as fully merged data. On completion of the calculations the program has (optional) graphical output enabling a series of plots to be produced and examined and, if required, to be saved in hard copy form on an individual plot basis. List of sections: The Control File Input and Output Files Running the Program Installation Notes The Log File Output Graphical Output The LSM Parameters The LSP Parameters The 'REFINE' Records Program Function Examples Some Hints about Input Parameters Reference Contact Address THE CONTROL FILE The program is run using a keyworded control file lscale.ctl (default) or another filename if so specified. This contains basically three classes of keyworded data: 1) The LSM parameters These are the Laue Scaling Module (LSM) parameters which hold the values of all the scaling factors which are applied to the raw integrated intensity data to give the corrected intensity data. These parameters are, where appropriate, updated during a program run. The LSM parameters are defined as a program independent set. 2) The LSP parameters These are a set of Laue Scaling Parameters (LSP) specific to setting up the required contol variables of the program LSCALE. These give some general details about the Laue data set and control many of the options to be used in the scaling process. 3) The 'REFINE' records After input of all the required LSM and LSP parameters, one or more records are given starting with the 'REFINE' keyword followed by a series of options. These refine records control which LSM parameters are to be refined, how the refinement is to be carried out and the output. The three sets of keyworded parameters are described in more detail in separate sections below. INPUT AND OUTPUT FILES The input files are: a) The control data file b) The Laue Data input reflection files These may be either .ge1 files or MTZ files as output by the program LAUEGEN and defined as part of the Laue Scaling Module development. The input program labels for an MTZ file are H K L PACK_ID PLATE XF YF LAMBDA I SIGI MULT MINHARM MAXHARM NOVPIX FLAGS The output files may contain merged or unmerged Laue data. For either option a MTZ file or a SHELX file or one of each of these may be output. For the MTZ files any reference data input will be copied to the output files. The possible files are as follows. 1) A reflection data file in standard MTZ format containing the merged normalised reflection data. The data items are H K L [FS SIGF] FP SIGFP DP SIGDP 2) An MTZ reflection data file containing unmerged but normalised data. The data items are: H K L M/ISYM BATCH I SIGI LAM [IS SIGIS]. 3) A data file containing the fully merged normalised reflection data (without anomalous) in SHELX format. (H K L F SIGF in format 3I4,2F8.2) 4) A data file containing the unmerged normalised reflection data (with lambda values) in SHELX format. (H L K I SIGI IPACK LAMBDA in format 3I4,3F8.2,I4,F8.4) Note: All the intensities written out to the MTZ- and /or SHELX-files are multiplied by 100.0 to avoid a loss of accuracy caused by the very small scale factors of the Polarisation-correction. Refinement details, statistics and results will be output on the screen if this wanted and/or redirected to a log-file lscale.log (default) or another file if so specified. Additionally, a file (outputfilename.par) is output containing the updated LSP/LSM keyworded parameters for the record and future input. RUNNING THE PROGRAM Use the command 'laue lscale' Parameters: DATA The control data file. The other input and output file names required are defined via the keyworded data. INSTALLATION NOTES The program runs on HP and SGI, the CCP4 library and XDL-view are required. It is advisable to compile the programs with the static-option. For large numbers of images or many parameters to be refined it might be neccessary to alter some array sizes in the lsm.inc file. If there is not enough memory, the program will stop and give the value to what the minimum array size for the refinement routine has to be set. THE LOG FILE OUTPUT The exact details of the log file output depend on the options selected when running the program but the output is broadly as follows: The log file starts with a listing of the input keworded LSP and LSM data. This is followed by details of the number and types of spots read in from the Laue input data files (.ge or MTZ). Then for each refinement carried out there is a section of output which details the parameters refined. After the first refinement tables of statistics are output for the data both prior to the refinement and after the refinement. These include tables of R-factors in wavelength bins and various overall merging R-factors. After subsequent refinements further tables of statistics are output. The R-factor tables are calculated with reflections selected according to the reflection output criteria. On completion of the refinement steps, details will be output of the multiples deconvolution if this was requested. Details of the reflections stored in the output files and finally an updated set of LSP and LSM parameters are listed. As part of the statistics, three types of Laue data merging R-factors are calculated. These are calculated as R = Sum(|Il-Imean|)/Sum(Imean) with sums over individual Laue reflections (Il = a Laue intensity) where Imean is calculated as follows for the three types. RFACT1 Imean = The mean intensity for all measurements (I+) and (I-) for the reflection (and all symmetry equivalents). RFACT2 Imean = The mean intensity for all measurements of the same sign (I+ or I-) for the reflection. RFACT3 Imean = The mean intensity for all measurements of the same sign (I+ or I-) for the reflection and with lambda values within 0.1 Angstroms. Where tables of R-factors between wavelength bins and between images are printed, the R-factor between bins 'n' and 'm' is calculated as Sum|(In-Im)|/Sum(In+Im) where the sum is over all reflections with measurements in both bins 'n' and 'm'. A reflection which has multiple measurements within a bin and has measurements in different bins will give several contributions to the R-factor. If there are N measurements in bin 'n' and M measurements in bin 'm' then there will be N*M contributions to the R-factor between bins 'n' and 'm' (and 'm' and 'n'). For the R-factor within bin 'n' containing N measurements there will be NC2 (=N!/2*(N-2)!) contributions to the R-factor. As an example to indicate how many contributions a reflection will make to a given R-factor, take the case where a reflection has 2 measurements in bin 'n' and 3 measurements within bin 'm'. The numbers of contributions are as follows: RFACT1 5 R(n,m) 6 R(n,n) 1 R(m,m) 3 Two further optional tables of R-factors between wavelength bins may also be output. In the second table only pairs of measurements of the same sign (I+) or (I-) are included and, in the third table, only pairs of measurements of the same sign and with lambda values within 0.1 Angstroms are included. The reflections used for the R-factor calculations and the scaling statistics use reflections selected according to the chosen restrictions for the data output. GRAPHICAL OUTPUT Unless requested otherwise, a menu will be displayed when the refinement calculations have been completed. This enables a series of graphical plots to be shown. Depending on whether an internal or external refinement is carried out, a reference set is given, multiples are deconvoluted or an absorption correction requested a more or less elaborate menu will appear after the refinement if the graphical window output option is choosen. But clicking at the various menu items diagrams of the statistical output will appear: - the error (Isc-Imean)/Imean before and after refinement - the normalisation curve displayed is the inverse of the wavelength normalisation factor) - I/sig(I) over lambda - resolution over lambda - the scaled Laue-intensities against the reference intensities if present - the deconvoluted Laue-intensities against the reference intensities if present - the plot of the absorption surface, i.e. a plot with the absorption correction valuesx1000 as a function of psi and nu. The values are plotted for the area of the images where data is present. They are normalised to the centre of the first image. - if a reference set is given there are various diagrams comparing the Laue intensities with the reference intensities in wavelength, intensity and resolution bins. (If more than one wavelengths range is given then the wavelength-analysis is only shown for the first range) When an individual plot is selected and displayed, it has an option available to produce a hard copy version in Postscript format by clicking on the PS-button at the diagram. THE LSM PARAMETERS The LSM parameters are described in more detail in the Laue Scaling Module (LSM) documentation but are summarised here for convenience. POL_Type type Polarisation correction type, 'xray' (default) or 'neutron' POL_Fac tau Polarisation factor (default=1.0) COVER_Fac ut Factor to take account of front sheet(s). exp(ut/cos(2*theta)) (default=0.0) PLATE_SCALE_r[][] almin almax kappa ups Plate to top plate scaling factors of the form kappa*exp(ups*lam**3) 'r' is a wavelength range number and almin and almax are the wavelength limits for that range. (default 0.0 0.0 1.0 0.0) IMAGE_SCale[] scal Relative scale factor for the image/pack. (default=1.0) IMAGE_BFac[] bfac Relative temperature factor for the image/pack. (default=0.0) REFSET_Scale scal Scale factor from reference data set. (default=1.0) REFSET_Bfac bfac Temperature factor from reference data set. (default=0.0) NORM_Lamref alam Reference wavelength for the wavelength normalisation curve in Angstroms (default=1.0) NORM_COEFF_r nord almin almax P{...) Wavelength normalisation parameters. 'r' is a wavelength range number, nord is the order for the Chebyshev polynomial and almin and almax are the wavelength limits for the range. P{...} are the Chebyshev polynomial coefficients (nord+1 values). (default = 0 0.5 1.5 2.0 for range 1 or 0 0.0 0.0 2.0 un-defined) for any additional wavelength range ABSCOR_Lamref alam Reference wavelength for absorption correction in Angstroms (default=1.0) ABSCOR_Mu C [A] or ABSCOR_Mu alam1 mu1 alam2 mu2 ... Parameters used in determining the wavelength dependent linear absorption coefficient used in the absorption correction. If two values are given, they are taken as being the coefficients 'C' and 'A' with mu = A + C*lam**3. If more than two values are given, they are interpreted as a table of pairs of lambda and mu values and the value for a particular wavelength will be interpolated linearly from this table. (must be in ascending lambda order) (default = 1.0 i.e. mu = lam**3) ABSCOR_Thick[] t0 General path length in mm through crystal at nu=0.0, psi=0.0 (local to the image/pack) (default=1.0) ABSCOR_Local[i] nord mord psimin_loc psimax_loc numin numax P{...} Local (image/pack based) absorption correction. P{...} are the Chebyshev polynomial coefficients excluding the P(0,0) term ((nord+1)*(mord+1)-1 coefficients) The minimum and maximum psi values are the local values relative to the pattern centre for the image/pack in question (in degrees). If a pair of angle range values are input as 0.0 0.0 then the LSM routines will calculate values based on the current data. (default = 0 0 0.0 0.0 0.0 0.0) ABSCOR_Global 1 nord mord psimin_glob psimax_glob numin numax P{...} or ABSCOR_Global 2 nord mord P{...} Q{...} Global absorption correction. The first term is a flag which determines the surface fitting option =1 Chebyshev polynomials, =2 Fourier coefficients P{...} are the Chebyshev coefficients or P{...}, Q{...} are the Fourier Coefficients. P(0,0) and Q(0,0) are not present. The minimum and maximum psi values are the global values relative to the pattern centre for the first image/pack in the dataset (in degrees). If a pair of angle range values are input as 0.0 0.0 then the LSM routines will calculate values based on the current data. (default = 1 0 0 0.0 0.0 0.0 0.0) THE LSP PARAMETERS These are a set of keyworded parameters used to control the LSCALE program and are used in addition to the LSM parameters. They are split here into sections for clarity but may appear in the input file in any order. Unless otherwise stated, a keyword is followed by a single value. If a keyword is given on a line of its own and not followed by any value then its default value will be reset. For image (or pack) or plate specific parameters the same general rules apply as in the Laue Data Module (LDM) parameter cases when image(pack)/plate specifications are not explicitly given. Sample related parameters TITLE string A title string to appear in the output. The default string is '** NO TITLE GIVEN **'. Maximum string length is 70 characters. SYMMETRY nsp This defines the crystal space group symmetry and is normally followed by the space group number or code though symmetry operators may also be defined. Its use is analogous to that in the Laue Data Module (LDM) where it is described in detail. A B C ALPHA BETA GAMMA a b c alpha beta gamma The cell dimensions in Angstroms and degrees. (default 0.0 0.0 0.0 90.0 90.0 90.0) The experimental setup SPINDLES inc Spindle angle increment between images. First image will be allocated spindle angle 0.0. (default = 0.0) SPINDLE_ANGLE angstart anginc Defines the spindle angles as a start angle (for the first image (pack)) and an increment for subsequent images (packs) (default 0.0 0.0) CTOF[][] ctof ... Crystal to image distance in mm (default = 0.0) IMAGE_SIZE xwidth ywidth Image size in mm (default 0.0 0.0) DETECTOR code Detector type 'flat' or 'cylinder' (default = 'flat') Defining the input images/packs NUMPACK n Number of images (packs) (default = 0) NPLATES[] n ... Number of plates/pack (default = 1) FILENAME_INPUT[] names ... Names of the input .ge1 or .mtz filenames for the Laue data. Maximum string length is 100. (The defaults are blank strings i.e. no file names) FILETYP_INPUT code File type of the input Laue data files. May be 'mtz' or 'ge'. (default = 'ge') The output file(s) FILENAME_OUTPUT name The name of the output scaled Laue data file. The type of file MTZ or SHELX is determined from the file extension which must be '.mtz' for an MTZ file or '.data' for a SHELX file. If neither is given then both types of file will be output and the program will automatically append extensionS of '.mtz' and '.data' for the two types respectively. Maximum string length is 50. (default is a blank string) OUTPUT_DATAOPT type Selections of 'merged' or 'unmerged' data for the output Laue data file. (default = 'unmerged') OVLD_OUT flag Output of overloads. Flag =0 do not output, =-1 output unconditionally (the default), >0 allow spots with up to this number of overloaded pixels to be output (only applies if MTZ Laue file input - for .ge1-file input, intensity overloads are never output) OUTPUT_SPAT flag Output of spatially overlapped intensities which have been integrated. Flag =1 output these (the default), =0 do not. OUTPUT_NEGINT flag Output negative intensities. Flag =1 (the default) output these, =0 do not. OUTPUT_SIGOUT sigout Sigma cutoff for output of intensities. Only output spots with intensities more than this number of sigma(I)'s. (default = 0.0) Reference Data File REFFILENAME name The name of the reference MTZ data file if present. (default = blank) Must be given if normalisation against a reference data set is to be done; may be given for statistics tables when an internal refinement is carried out. Maximum string length is 100. REFMTZ_FILTYP code Normally the reference data file is a standard MTZ format file. Special options are allowed for files containing sign and/or wavelength separated data. The options are 'standard' (the default), 'lambda_sep', 'sign_sep' and 'lambda&sign_sep'. REFMTZ_DLAM dlam For lambda separated data, reflections are considered to match if their wavelengths are within this value of the wavelength of the reference reflection. (default = 0.1 Angstroms). REFSET_RESCALE scal Divide the input reference intensities by this factor to put them on approximately the same scale (order of magnitude) as the Laue intensities. (default = 10.0E05) REFMTZ_FILEASSIGN FP=name SIGFP=name MTZ assignments for the reference data for FP and SIGFP R-factor tables RFAC_OUTPUT Controls the output of lambda binned R-factor tables. =0 omit R-factor tables =1 RFACT1 table (default) =2 RFACT1 and RFACT2 tables =3 RFACT1 and RFACT3 tables =4 RFACT1, RFACT2 and RFACT3 tables Selection of reflections SIGN_SEP flag Treat Friedel pairs as two separate reflections =1 yes, =0 no (default) SPATOV flag Include spatial overlaps =1, yes =0 no (default). OVERLD flag Include intensity overloads =1 yes, =0 no (default) SIGREJ_EX sigrej_ex Exclude reflections from use in the refinement if they have intensity values less than sigrej_ex*Intensity (default = 0.0) CUTOFF cutoff Outlier cutoff value (outliers with (|Iscl-Imean|/Imean) > cutoff value are excluded) (default = 0.0, i.e. no outliers are excluded) OMIT List of images (packs), and/or ranges of images (packs) to be omitted e.g. 5, 7-9 will omit images (packs) 5, 7, 8 and 9. Absorption Correction ABSOPT flag Flag for modelling type of absorption surface =1 (the default) model the generalised path length, =2 model a normalised absorption scaling factor. Refinement cycles NCYCLES ncyc The number of cycles to be carried out when carrying out a refinement in cycles (default = 3) Deconvolution of multiples MULTIPLES flag Carry out deconvolution of multiplesa flag =1 yes, =0 no (default). THE 'REFINE' RECORDS The REFINE keyword contains the information about which parameters have to be refined, in which order the refinement is to be carried out, options for outlier exclusion, and whether the program runs in batch mode or with the XDL-view output windows. It is also possible to choose default settings (described below). Options: DEFAULT The default option refines the following parameters together: IMAGE_SCALE NORM No outliers are excluded and the program runs with the xdl-view output. All following options will be ignored. TOGETHER or CYCLE TOGETHER: Refine all parameters together. CYCLE: Refine parameters in cycles, i.e. one parameter after the other within each cycle, the number of which can be input with the keyword NCYCLES. The parameters will be refined in the following order: Plate scaling factor Image scaling factor Image temperature factors Normalisation curve Reference set scaling factor Reference set temperature factor Global absorption correction Local absorption correction Refinement with outliers excluded If neither of the 2 options is given a warning message will be printed and the default 'CYCLE'is used. NODISPLAY This keyword allows to choose between the xdl-view windows output of statistic or to run the program entirely as a batch job (the log-file contains all the refinement information). If this keyword is omitted, the xdl-view output is enabled. PLATE If the keyword is given then the plate scaling parameters for all plates with valid data present (i.e. for all packs) will be refined, if the keyword is omitted, no plate scaling parameters will be refined and the plate scaling factor for the images will be set to 1.0. IMAGESC Refine image(pack) scaling factors for all images(packs) (except image/pack 1). If omitted, no image(pack) scaling factors will be refined, except if the keyword 'EXTERNAL'is given and reference data are present. IMAGEBF Refine image temperature factors for all images(packs) (except image/pack 1). If omitted, no image(pack) temperature factors will be refined, except if the keyword 'EXTERNAL'is given and reference data are present. NORM Refine normalisation curves for all defined ranges. GLOBABS Refine global absorption correction, if omitted no global absorption correction will be refined.(If both GLOBABS and LOCABS are given a global absorption correction will be applied.) LOCABS Refine local absorption correction, if omitted no local absorption correction will be refined.(If both GLOBABS and LOCABS are given a global absorption correction will be applied.) EXTERNAL If the EXTERNAL keyword is given then the refinement will be against a reference intensity from an external data set (for example monochromatic) rather that against the internal data (i.e. the mean intensity values of a group of equivalent reflections). Reference data have to be present and the reference scale and temperature factures will be refined. Default is an internal refinement. EXOUT If EXOUT is given, the program carries out an additional refinement at the end with all spots with |Isc-Imean|/Imean>CUTOFF excluded. This excludes outliers with a low sigma, often does the R-factor improve considerably with the exclusion of very few spots, but be carefull, the outliers won't be written to the output-files. PROGRAM FUNCTION The program LSCALE scaling and normalizes raw integrated Laue intensity data to yield fully corrected structure amplitudes. It unites and improves the fuctions of the older programs LAUENORM and LSCALE formerly used for Laue data processing. The wavelength normalisation curve is calculated with Chebyshev polynomials. It can be derived internally from the Laue data using symmetry equivalent data recorded at different wavelengths or as an alternative by scaling the Laue data to a reference e.g. monochromatic set of data. The wavelength normalisation can be carried out for up to five wavelengths-ranges. The program also enables a wavelength and position dependent absorption correction to be calculated by two-dimensional Chebyshev-polynomials. Harmonic multiples data may be deconvoluted again making use of symmetry equivalent data recorded at different wavelengths. In order to deconvolute as many multiples as possible the wavelength range will be temporarily expanded over the full data range. When there is a measured single reflection in the equation system it will be used for deconvolution, but put in the output file with its old value. The program provides flexibility in the control of the refinement procedure. Refinable parameters can be refined all together in the least squares refinements or one after the other in any order. The refined scaling parameters are saved for re-use as required. The scaled data may be output in a number of formats as unmerged or as fully merged data. On completion of the calculations the program has (optional) graphical output enabling a series of plots to be produced and examined and, if required, to be saved in hard copy form on an individual plot basis. EXAMPLES Example 1: This is a simple control file for the processing of 24 Laue images. The image scale and temperature factors and the wavelength normalisation curve will be refined internally, no absorption correction will be carried out. All parameters will be refined together. The wavelength normalisation curve will be calculated between 0.480A and 1.3A. Spatially overlapped reflections will be included and the multiples deconvoluted. Only spots with I> SIGREJ_EX*sig(I) will be used in the refinement and an additional refinement excluding all spots with |Isc-Imean|/Imean greater CUTOFF will be carried out after the normal refinement. Unmerged data excluding all spots with I SIGREJ_EX*sig(I) will be used in the refinement. Unmerged data excluding all spots with I