Automated Data Reduction

Sourceforge project page.

New! I have now set up a mailing list for xia2 on sourceforge. If you would like to subscribe to this - where announcements of new versions of xia2 and problems can be discussed, please go to the list page. Also, if you are interested in keeping up-to-date with what is going on in xia2 development take a look at the blog.

xia2 is a new automated data reduction system designed to work from raw diffraction data and a little metadata, and produce usefully reduced data in a form suitable for immediately starting phasing and structure solution, e.g. through Mr BUMP or your favourite experimental phasing suite.

The following steps are performed (although not strictly in this order)

Integration with feedback and complex lattice selection stuff
Reindexing to standard setting for spacegroup (e.g. P 2 21 21 to P 21 21 2)
Scaling together of all data to maximise anomalous signal
Estimation of number of molecules in ASU and solvent fraction based on sequence information (if provided)

There is a lot of fun stuff which goes on in there - run it to find out more...

The system is designed to cope with MAD data and data measured in multiple passes, or a combination of the two. The input is a therefore a little complex, but consists only of information you should have to hand anyway - here is a simple example, a more complex example can be found below.

Download

The current version, 0.3.6.0 can be obtained by following these links:

All of the files are the same, simply packed for different platforms.

Installation

Since everything is Python using cctbx, the only requirement is to export XIA2_ROOT to point to the directory where xia2 was unpacked (including the xia2-0.3.6.0 bit) and then source $XIA2_ROOT/setup.(c)sh.

Input

There are two ways of running xia2 - with and without an input file, an example of which follows below. If you just run

xia2 /my/data/are/here

xia2 will do something sensible - in this case process all of the data, scale all measurements as if they are from a single crystal and merge the data from each wavelength separately. If only one wavelength is present xia2 will assume that the data are a native data set - to separate anomalous pairs provide a heavy atom (at this time it doesn't matter what it is...) i.e.

xia2 -atom se /my/data/are/here

Other options are (type just xia2 to get this list)

Command-line options to xia2:
[-2d] or [-3d] or [-3dii]
[-parallel 4] (say, for XDS usage)
[-resolution 2.8] (say, applies to all sweeps)
[-freer_file free.mtz]
[-quick]
[-atom se] (say)
[-reversephi]
[-migrate_data]
[-beam x,y]

Running ths way some assumptions are made:

All images are from one crystal only.
The scan, if present, was the one used to select the wavelengths for data collection. If more than one is present, the alphabetically latest one will be used.
The sequence file, if present, should be in the one letter form and all comment lines should begin with a character not in A-Z. Again the alphabetically latest sequence file will be used.
If the sequence file is provided, and the atom is "se", then xia2setup will assume that you are doing a SeMet experiment and will provide an appropriate number of atoms per monomer - though you will still have to uncomment this to include it, in case it has guessed wrong.

If you want to combine data from a number of crystals in the same .xinfo file, then you will need to copy out all text from BEGIN CRYSTAL to END CRYSTAL from one .xinfo file to the other.

You should then load automatic.xinfo in your favourite editor, and check that the sequence looks correct and that the names are sensible as well as checking that the epoch numbers are set correctly and also that the wavelengths and beam centres are correct. If you provided a heavy atom there is a place to say how many to look for.

Finally, if you have labelit installed xia2 will run this to update the beam positions. If this happens, you will see a comment to this effect above the BEAM records in the sweeps.

The other mechanism for running xia2 is via a xinfo file, which explains the layout of the data set to xia2 explicitly. This is helpful if you wish to only process a subset of the measurements, or want to process data for an RIP experiment. A simple example of xinfo file follow below, and more complex examples can be found here:

More examples can be found in $XIA2_ROOT/examples. Remember - you need to tell xia2 where to find your data! Answers to frequently asked questions can be found here.

! This is a demonstration .xinfo file which illustrates how to cope
! with a simple case - this example is a native cubic insulin data
! set measured on 14.2 at the SRS

BEGIN PROJECT DEMONSTRATION

BEGIN CRYSTAL INSULIN

BEGIN AA_SEQUENCE

! this is only really needed at the moment for assessing the solvent
! content and number of residues in the asu

GIVEQCCASVCSLYQLENYCN
FVNQHLCGSHLVEALYLVCGERGFFYTPKA

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE

! this doesn't have to be here - if it is
! not included then the values from
! the image headers will be used - however
! if it is there then it should
! be correct!

WAVELENGTH 0.979000

! in here you can also have
! F' value
! F'' value

END WAVELENGTH NATIVE

BEGIN SWEEP NATIVE
WAVELENGTH NATIVE
IMAGE insulin_1_001.img

! you will probably need to change this -
! this is the only thing which
! you will need to change for the
! demonstration data set

DIRECTORY /media/data1/graeme/demo/

! additionally you can add the following
! information - if it is wrong in the headers
! BEAM x y (mm)
! DISTANCE z (mm)

! this describes the order in which
! the sweeps were collected -
! it usually comes from the image header
! if that information is in there
! EPOCH 5

! you can also add this to only reduce
! a subset of the data
! START_END 1 30 (image numbers)

END SWEEP

END CRYSTAL INSULIN

END PROJECT DEMONSTRATION

Demonstration Data: Cubic Insulin

This uses test images which are available from the links below. Thanks to John Cowan for providing this test data!

Platforms

The following platforms are supported (see notes at bottom of page for platform specific advice.)

GNU/Linux on i386 (and 64 bit extensions)
OS X 10.4 i386 and Power PC G4/G5
Microsoft Windows 2000/XP (Vista doesn't work - problem with the mosflm SPOTOD file - currently investigating)

Supported Detector Types

The following detectors are supported:

ADSC CCD detectors
MAR CCD detectors
MAR 345 image plates
RAXIS IV image plates
Rigaku Saturn CCDs (Binned and unbinned)
Pilatus 6M detectors

If you have another detector class which needs support, please get in touch.

Requirements

The only software which is actually required is CCP4 6.0.1/2 and Python 2.4.x. On Windows the PyWin32 module is also required, to provide additional process control functionality. If you are using this on a Windows platform and wish to be able to use the integration functionality, you should get in touch with Harry Powell for a custom build of Mosflm.

In addition the autoindexing program Labelit is supported but the system will work without it. If it is installed (and it should be a moderately recent version) then it will be used for autoindexing.

License

This software is distributed under the BSD license for all users. Everyone who uses this software is invited to join the xia2bb mailing list. Finally, if you use this software in solving a structure which is published, please acknowledge it! Thanks!

A copy of the license is available here .

This software depends apon CCP4 and will also make use of Labelit - users are reminded that it is their responsibility to have properly obtained licenses for this software.

Usage

Once the software is installed, the first thing you need to do is prepare a .xinfo file (example above) to run from. This contains a description of the experiment, which will allow xia2 to decide how to handle your data and what information to put in the MTZ headers.

Essentially the input file just needs a description of the data which were collected from each crystal - for instance, if you measured three MAD wavelengths, a peak, inflection and high remote, then there should be three WAVELENGTH blocks in the file, containing the fp, fpp and wavelength values. The images that you measured from these should then be described in the SWEEP block, which is then assigned to a wavelength. This structure simplifies handling of multiple passes contributing to one wavelength. In the reduced output, the wavelength names will correspond to MTZ datasets. If you are interested in keeping the harvesting information, and you have more than one crystal to process in a single xia2 run, you will need to ensure that the WAVELENGTH names are globally unique within the input file.

The program is actually run with:

xia2 -xinfo project.xinfo

And all data reduction is performed in the working directory, with the a directory structure CRYSTAL/WAVELENGTH/SWEEP. Interesting log files can be found in the CRYSTAL/scale directory.

Acknowledgements

Development of xia2 is supported by the BBSRC e-HTPX grant and the EU framework 6 BioXHit initiative. xia2 makes extensive use of Phil Evans's program Pointless, and I would like to thank him for the extensive modifications made to the program. I would also like to thank the US structural genomics JCSG group for providing their extensive collection of raw diffraction data to methods developers. Finally I would like to acknowledge CCP4, for without their software this project would go nowhere!

The More Complex Example

This example has two crystals, one of which is selenomethionated, with two wavelengths (inflection and low remote) measured for the derivative and two native sweeps measured for the native sample. Lines beginning with # or ! will be treated as comments.

BEGIN PROJECT TS01
BEGIN CRYSTAL 12847

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN WAVELENGTH NATIVE
WAVELENGTH 0.99187
END WAVELENGTH NATIVE

! high resolution native pass

BEGIN SWEEP NATIVE_HR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_4_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_HR

! low resolution native pass

BEGIN SWEEP NATIVE_LR
WAVELENGTH NATIVE
BEAM 109.0 105.0
IMAGE 12847_5_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.2.1/20050121/collection/TM0892/12847/
END SWEEP NATIVE_LR

END CRYSTAL 12847

BEGIN CRYSTAL 13140

BEGIN AA_SEQUENCE

MKVKKWVTQDFPMVEESATVRECLHRMRQYQTNECIVKDREGHFRGVVNKEDLLDLDLDSSVFNKVSLPD
FFVHEEDNITHALLLFLEHQEPYLPVVDEEMRLKGAVSLHDFLEALIEALAMDVPGIRFSVLLEDKPGEL
RKVVDALALSNINILSVITTRSGDGKREVLIKVDAVDEGTLIKLFESLGIKIESIEKEEGF

END AA_SEQUENCE

BEGIN HA_INFO
ATOM SE
NUMBER_PER_MONOMER 5
END HA_INFO

BEGIN WAVELENGTH INFL
WAVELENGTH 0.979741
F' -10.0
F'' 3.2
END WAVELENGTH INFL

BEGIN WAVELENGTH LREM
WAVELENGTH 1.019859
F' -2.6
F'' 0.55
END WAVELENGTH LREM

BEGIN SWEEP INFL
WAVELENGTH INFL
BEAM 108.7 102.0
IMAGE 13140_1_E1_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

BEGIN SWEEP LREM
WAVELENGTH LREM
BEAM 108.7 102.0
IMAGE 13140_1_E2_001.img
DIRECTORY /Volumes/Arthur/JCSG Data/1vr9/data/jcsg/als1/8.3.1/20050105/collect/TM0892/13140/
END SWEEP

END CRYSTAL 13140

END PROJECT TS01

This is a particularly complex example, but it illustrates the capability of the system well. There are data from two cystals, one native and the other selenomethionated. The native set is measured in two sweeps. Currently (version 0.3.5.1) no attempt is made to intelligently combine data from multiple crystals, however this is on the to-do list.

Release Notes

Changes since 0.3.5.2: dealing with huge data sets

If single sweep, do not rebatch.
Replace CCP4 Mtzdump with CCTBX-powered equivalent.
If reindex for changing spacegroup name but not indices (i.e. no REINDEX= card) then use CCTBX-powered replacement.
Cleaned up repository by pulling xia2core into xia2/core.
Patched against dxtbx being included in Phenix. (temporary)

Changes since 0.3.5.1: CCP4 patch release

Resolution limit calculation now delegated to resolutionizer.
Twinning test now uses CCTBX code not sfcheck.
PythonDriver added to support execution of Python code as separate process.

Changes since 0.3.5.0

-hdr_in mode - pass in text file containing image header. Can also generate this file with xia2 with -hdr_out example.hdr
-xparm_ub mode for XDS processing only, provide a UB matrix (XDS XPARM file) to match the orientation to: primarily useful if you record data from several regions on a sample, and the sample has close-to symmetry (i.e. you almost certainly don't need this)
Removed usage of BINSORT_SCR as this is being deprecated by CCP4.
Allow Phil overrides on selection of autoindexing solutions, where indexing can give large variations in unit cell constants (e.g. low resolution virus data)
Properly set the resolution in CAD copying free R flags.
Allow access to profile grid size through the Phil interface: xds.parameter.profile_grid_size = 13, 13
Implement TRUSTED_REGION through Phil as suggested by Luca Jovine
Updates to process simulated data from SIM_MX
More measures of anomalous signal in output.
Corrected citations in output
Replaced TRUNCATE with CTRUNCATE.
Replace sfcheck with E^4 calculation from cctbx.
Exclude grid-scan data when running automatic setup.
If XDS refinement not performed, exception not warning.
If -3d and sweep narrower than 10 degrees, use XDS II indexer.

Changes since 0.3.4.0

Additional information in output: correct directory for BINSORT_SCR, free space in working directory, warns if less than 200MB in BINSORT_SCR directory.
EXCLUDE keyword added to XINFO SWEEP block to allow specific resolution ranges to be excluded in processing, contributed by Andrew Perry at Monash University.
Include the resolution limits at the XSCALE stage rather than simply in the merging.
Support for 2x2 binned ADSC Quantum 270 detector.
Support for '.' in file names.
Support for NOIR1 detector on ALS 4.2.2
Smarter 3dii mode - if 3di style indexing will give better answer, it will do that...
Use ctruncate in place of truncate (finally!)

Changes since 0.3.3.4

Use Phil to override XDS parameters.
Use Pointless in place of Reindex; Aimless in place of Scala; no limit to number of batches which can be processed.

Changes since 0.3.3.3

Fixed bug in polarization correction vector for Pilatus data.
Fixed bug relating to assigning spacegroup on command-line.
Begun work on supporting Pilatus 300K for small molecule crystallography. Have now successfully processed data from Diamond Light Source beamline I19 EH2. N.B. Only works -3dii!

Changes since 0.3.3.2

Many bug fixes as a result of implementing a comprehensive testing `scheme.
Support for full cbf format, making use of cbflib Python bindings pycbf now included in cctbx releases.
Merged functionality from XDSIntegraterR into XDSIntegrater, removed duplicate code. Will make for more maintainable suite in the future.
Improved 2d pipeline scaling, now tests 8 permutations.
Improved handling of twinned data for multiple sweeps.
Fixed occasional bugs with multiple sweeps, reindexing trigonal data sets with XDS.
Removed "old" scaling modes, 2d and 3d, as these were unhelpful.

Changes since 0.3.3.1

-xparallel -1 option added which will split your data into 30 degree chunks and process with forkintegrate Note well you are responsible for setting up forkintegrate correctly!

Changes since 0.3.3.0

Handle division in updated CCTBX.
Squash bug with -3d and -resolution.
Squash bug with resolution and -2d, result slightly different to that reported.
Trap weirdness with XDS reindexing, causes problems sometimes trying to process in C2 from P222 not P2 as you would hope!
Adjustments to cope with excludively narrow sweeps i.e. all 3 image sweeps.

Changes since 0.3.2.0

Support for new XDS build.
Small bug fix / improvement in XDS indexer implementation.
Capability to ptovide refined experimental geometry via XDS GXPARM.XDS - useful for polycrystal data reduction.

Now check what you tell the program, viz:

gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense

and complain if you type in something xia2 does not understand.

Removed a lot of cruft from the xia2 code base in preparation for some more substantial refactoring.
Found bug which meant that the labelit beam centre computed in the setup phase was not used - mostly this is not important, unless you are using -3dii.
Added -ice command-line argument, which will exclude measurements from regions which are typically where ice rings land. Will need to add more subtle mechanism which will allow specific regions to be excluded.
Specifying resolution with 3d(r) pipeline now works correctly.
Removed XDS version check by default, which was annoying every new year. If you would like to check that the XDS version is explicitly supported, add -check_xds_version to the command-line.
Added table of scala runs to sweep name in the debug output, helpful for reviewing in polycrystal data reduction cases.
Mosaic spread used from Mosflm is now average of all images, not the first or last image.
Fixed header reading for new Pilatus instruments at DLS and elsewhere.
Added capacity to pass in known globally postrefined experimental setup via -xparm GXPARM.XDS, only useful with XDS processing pipelines.
Removed use of ccp4 printpeaks tool, which crashes on new pilatus images, replace with Mosflm for the moment. Move to replacement with labelit code at later stage planned.

Changes since 0.3.1.7

Now report the min, mean, max mosaic spread (according to the program definitions after integration of each sweep.
Resolved problems with overriding environment.
Added -serial pun for -parallel 1.
Allow resolution limit to be assigned on a per-sweep basis.
Handled corner case with images 200 - 250 (say) giving REBATCH error.
Added full 8-way search for scaling models for 2d pipelines.
Partial support for two-theta offset data collection (tested with Rigaku X-ray equipment and Diamond I19 data.)
Added -min_images keyword which is helpful when analysing virus data with e.g. 4 images / sweep.

Changes since 0.3.1.6

Changes suitable for working on microcrystals - reachable through a -microcrystal command line flag, and -failover is probably helpful too.
If lattice or spacegroup specified, do not run tests.
2dr and 3dr now the default pipelines - use 2dold or 3dold if you need to get to the old pipelines.
Fixed selection of known lattice in IDXREF reindexing.

Changes since 0.3.1.0

Fixed bug where the resolution limit was not reset in an indexer solution elimination.
Added support for XDS from Dec 09.
Fixed some of the issues found in the resolution limit determination when processing low resolution data.
Fixed use of I/sigma limit assignment.
Added command-line control of individual indexer, integrater, scaler.
Repaired regular expression for image matching, to cope with images where there is e.g. 3.5 in the template. This was previously misinterpreted as an image name of the form setup.NNNN.
Implemented new resolution limit pipelines, based on new merging code to give more robust control: -2dr and -3dr (recommended.) Fine control over the choices can be made using the -isigma, -misigma, -completeness and -rmerge command line options.
Added interactive indexing mechanism, where user can assign images to use for autoindexing, e.g.:
```
Existing images for indexing: 1, 90, 180
>1, 60, 120, 180
New images for indexing: 1, 60, 120, 180
        
```
Assign -interactive on the command line (Mosflm and Labelit indexers)

Changes since 0.3.0.6

Added -xparallel command line option to allow use of forkinitegrate on a cluster with the 3d pipeline.
Allow for xia2html and (beta) ISPyB output (latter more useful for synchrotron sources, former handy if you want a report.
Smart scaling mode switched on by default.

Chef now run by default, for radiation damage analysis, e.g.

Group 1: 2-wedge data collection
Group 2: Single wedge
Significant radiation damage detected:
Rd analysis (12287/LREM): 15.17
Rd analysis (12287/INFL): 7.89
Rd analysis (12287/PEAK): 12.82
Conclusion: cut off after DOSE ~ 608.8

Blank images now handled more gracefully.
Overloaded and blank images now correctly reported in the summary output, along with abandoned images for the processing with Mosflm. Also postrefinement results.
Substantially cleaned up the program output.

Changes since 0.3.0.5

No lattice test mode added (you can guess the command-line option) for tricky data sets where this perhaps falls over.
Fixed side effect of changes which allowed setting of unit cell etc. - lattices were no longer eliminated, failed complaining can't eliminate only solution.
Tidied up the generation of output files and so on - now only have the main log file and the debug trace.
Added copy of the command line to program output.

Changes since 0.3.0.4

Now able to specify number or fraction of reflections assigned to the free set, rather than the default 5 percent.
Added trap for sdcorrection not being refined in XDS correct if multiplicity rather low. Unusual case.
Assigning a freer_file by definition sets this as an indexing reference and also copies the spacegroup assignment. This is what you would expect!

Changes since 0.3.0.3

User now able to assign cell constants: use with great care, as there is little in the way of nonsense trapping. This may however be used to handle cases where the solution you want is monoclinic with a pseudo-orthorhombic lattice, where the default indexing would select a different setting. This will not currently work with a Mosflm indexer. Usage is:
```
-cell a,b,c,alpha,beta,gamma
```
and the correct symmetry should also be assigned. Can also assign USER_CELL a b c alpha beta gamma in the xinfo file.

Changes since 0.3.0.0

Now automatically determine the number of available processor cores.
Implemented check for centring of crystallographic basis from autoindexing.
Started to use CCTBX - this will be bundled with the release from now on...
Now runs "smart scaling" i.e. will customise the scaling model used to the data. This can seriously improve the xia2 run time in cases where some of the default scaling models do not converge.
Now also includes running of CHEF for radiation damage, which will slice and dice your data into dose groups, then run a correctly time sequenced radiation damage analysis. N.B. for data measured in "dose mode" the doses will be scaled to < 1,000,000 to ensure that the output is tidy.
Corner case of running XDS pipelines from data from a Rigaku setup. The low resolution was previously calculated to be 0.0! D'oh!
Added access to reference reflection file functionality to the command line - -reference_reflection_file foo.mtz.
Fixed the use of the -image command-line option. Now works.
Removed all of the binaries from the distribution, so now you need to be using CCP4 6.1.0 or later!
Will correctly handle reindexing with the 3d pipeline (XDS/XSCALE) with trigonal spacegroups and multiple sweeps.

Changes since 0.2.7.2

Fixed merging statistics bug - I/sigma output were very slightly different.
Now extend FreeR column if copying from lower resolution input file.
Detector limits now correctly specified around the detector centre for Mosflm, rather than the beam centre. Only useful if your beam centre is a fair way from the image centre...
Parallelised integration with Mosflm - divides the sweeps into an appropriate number of chunks and then sorts together the resulting reflection files. -parallel N, remember.
Added a putative 'small molecule' mode, which will autoindex with Mosflm from a modest number of images, for smaller molecules where Labelit is unhappy with the number of good Bragg reflections.

Changes since 0.2.7.0

Fixed to work correctly with XDS and Rigaku Saturn/RAXIS detectors.
Better determination of resolution limits.
Correct merging of data from XSCALE (needs to invert the I=F*F scale factor...)
Fixed output of scalepack unmerged output using 3d pipeline.
Use fewer frames for background calculation with 3dii pipeline - speeds things up a fair amount.
Write chef output files with correct SD correction parameters.
Print SD correction factors for scala runs - if these are bigger than about 2 there is something properly wrong. Also print the same from XDS CORRECT step.

Changes since 0.2.6.6

All command line options now echoed to the standard output if -debug selected.
Command line options now copied to .xinfo file when operating in automatic mode.
XDS: low resolution limit determined from spot list from IDXREF.
Now include explicit python version check in xia2 main program.
High resolution limits for integration now assessed from Wilson plot rather than integration program log output.

If no images are given, a more helpful error message is produced than before, viz:

------------------------------------------
| No images assigned for crystal DEFAULT |
------------------------------------------

If images are not readable (i.e. a permissions problem) then warnings will be sent to the debug channel (switch on -debug for more information.)
If in automatic mode an image is given in place of a directory, xia2 will now complain in a more helpful way.
Resolution limits will now be based on analysis of the reflection files rather than the program output, since this will generally give a more helpful answer...
Now correctly set the reindex matrix for running correct with multiple sweeps of data for non primitive matrix (subtle bug, thanks to Kay Diederichs for helping to fix this one.)
Is images are missing from a sweep, xia2 will now tell you this before starting processing rather than giving a strange error - if the images are unreadable you will also be told.

Changes since 0.2.6.5

Now supports CCP4 6.1 (forthcoming release) though you may need to add XIA2_CCP4_61=1 to your environment. This release should include xia2.
Added -user_resolution keyword, also support for this on a per-wavelength basis in the xinfo file.
Added support for reverse-phi data collection - either add reversephi to the xinfo file or -reversephi to the command line.

Changes since 0.2.6.4

Now handles complex image names with e.g. + in them correctly.
New option for processing
```
-3dii
```
added - this will run the 3D pipeline (XDS, XSCALE) using all images for spot picking and autoindexing - useful for properly tricky cases.
Now correctly propogate the profile information to Mosflm when not using Mosflm for autoindexing - this is done by running a "fake" autoindexing task.
Use the raster information for the cell refinement test, and changed the selection of images for this.
Added support for the output of BioXHIT XML in the LogFiles directory - for project tracking.
Now copy the XDS log files ("LP" files) into the LogFiles directory - as always these are the most up-to-date versions from processing.
Added support for latest release of XDS (June 2, 2008.)

Changes since 0.2.6.3

Added possibility to give a reflection file with an existing FreeR_flag column, which will be copied to the output MTZ:
```
xia2 -freer_file free.mtz
        
```
or...
```
FREER_FILE free.mtz
        
```
in the xinfo file (in the same way as the reference reflection file - indeed, this may be the same file...)
Added command line control of the spacegroup - note well that this means that the data WILL NOT be reindexed to a standard setting.
Added support for ADSC Quantum 270.
Intelligent selection of autoindex threshold for Mosflm.
Added support for latest XDS version.
Added support for pilatus 6M mini CBF images.
Fixed problem with cell refinement giving negative mosaic spreads sometimes for 2d pipeline.

Changes since 0.2.6.2

Now works for images with long image numbers, as typically recorded on Rigaku X-Ray sets.
Signed off for operation with Rigaku Saturn and RAXIS IV detectors.
Now works with no input, e.g.
```
xia2 -project TG6623 -crystal X77788 -atom se /my/images/are/here
        
```
However this relies on your image headers being accurate and the images having some kind of recognisable format...
Repaired operation on ppc and intel macs - added required libraries to the installation and reset the library paths appropriately.
Slightly improved error reporting from xinfo file errors.

Changes since 0.2.6.1

Now carefully select the images to use for cell refinement based on the orientation of the crystal lattice.
Fixed numerous bugs to do with the naming of the detector class which changed in the previous version.

Changes since 0.2.6.0

Fixed bug where if the distance was read incorrectly from the image header (or was wrong therein) XDS would get the wrong value even if you had put the correct value in the .xinfo file.
Now "unroll" the unmerged reflections from XSCALE and then merge them in their original sweeps in Scala. This should give a useful Rmerge vs. time plot.
Including updated versions of Pointless and Scala.
Includes new and more robust handling of pointgroups, lattices and unstable refinement of parameters during integration.
Include support for a reference reflection file, which will ensure that the reflections are indexed in the same way and with the same spacegroup - useful for mutants.
Now always use three wedges of images for the cell refinement with Mosflm as this makes the lattice elimination more reliable.
If one of the images in a sweep is broken (e.g. corrupted) xia2 will tell you more helpfully rather than just crashing.
Now correctly read distance from Mar 165 images. Thanks to Francois Remacle for this fix to DiffractionImage.
Fixed the use of xia2 -3d with RAXIS IV detectors.
Use more images for Mosflm autoindexing (three instead of two) as this gives uniformly better results.

Changes since 0.2.5.2

Now works for highly incomplete data sets, so long as you have either 4 times the mosaic spread or 8 images, whichever is the larger. This is to allow pointgroup identification from e.g. the first 10 images in a data set - designed to assist in synchrotron / lab source data collection.
Now include the latest Mosflm binaries in the extras package - these are often better than those in the CCP4 distribution, as they come directly from Harry Powell's web page.
Added -ehtpx_xml_out option which will write out marked up metadata about the data reduction, for inclusion in the e-HTPX data reduction portal and, perhaps, other automated systems. Usage:
```
xia2 -ehtpx_xml_out project.xml
        
```
If data are very incomplete (e.g. less than 50% complete) then don't try to refine the error parameters. This is both incorrect and a waste of time.
Renamed the output reflection files - these are now in DataFiles directory with names like "PROJECT_CRYSTAL_free.mtz." This makes them a little easier to identify.
Harvesting (deposition) files now in subdirectory of Harvest rather than being spread around the place. These will have names Harvest/DepositFiles/PROJECT/WAVELENGTH.scala etc.
Have now fixed xia2setup so it works much more sensibly - the decision about when a new sweep starts was a little broken (rounding errors issue) now fixed!
Added -parallel flag to work when using XDS for data reduction.
Added XDS support! This is however much less mature than the support for Mosflm/CCP4. It is also worth noting that this relies on many CCP4 tools.
Added selection of the "pipeline" to use through a command line option, either -2d for mosflm/ccp4 or -3d for xds/xscale though the latter still uses a number of CCP4 programs.

Changes since 0.2.5.1 - big changes in bold

Process data as native if no heavy atom information, only one wavelength specified and no anomalous scattering form factors provided. Otherwise separate anomalous pairs for scaling.
Added a -quick option to the xia2 command line, which will cut out many of the data reduction optimisation steps (not refine detector parameters, resolution or scaling parameters) but will still include everything else. This is to allow quick characterisation of data at the beamline, and perhaps map calculation?
Correctly print the unmerged scalepack file name for single wavelength data.
Perform radiation damage analysis using R merge and B factor as a function of time - note that this is not batch. This is likely to have interesting side-effects.
If time of data collection not recorded in header but the time stamp for the images are correct (e.g. you have not moved them since collection) then the command line flag
```
-trust_timestamps
```
can be used, which will use the time stamps on the image files to analyse things like radiation damage.
If you are running this on a cluster, have added an option to migrate the diffraction data to a local disk (e.g. /tmp.) To do this add
```
-migrate_data
```
to the command line - the data will be removed from the local disk once the processing is finished.
Copy all final reflection files to a DataFiles directory.
Now, when something goes horribly wrong, just write the actual message to the screen and the horrible stack trace gunk to a file called xia2.error. This should be sent in for error reports!
Remerge individual wavelengths to get the merging statistics to the most appropriate resolution limit, rather than the furthest extent of any data, for the summary table.
If GAIN included in SWEEP block, will be used as the default value for integration. E.g. GAIN 0.25.

Changes since 0.2.5

Fixed problem with spacegroup R3:H (naming convention problem - there's a surprise.)
GAIN estimation temporarily removed.
Pointgroup evaluation fixed - found a major gremlin in there

Allow environment variables and tilde in DIRECTORY token:

DIRECTORY $DATA/example or ~/data or %DATA%/example (win32)

Now tracks the citations for the programs used, writing them to the standard output in plain text and to xia-citations.bib in BIBTeX format.
Removed [XIA2] tokens from standard output - the reason for having this there no longer applies.

Changes since 0.2.4

xia2setup will now add the f', f'' values if a scan is available and has been processed by chooch
GAIN now estimated by diffdump - enabling future parallelisation of the integration stage.
For detectors formed as a mosaic of a number of tiles, with "gaps" in between, now mask those gapped areas in Mosflm to reduce the number of bad reflections.
Now has a fix for the ongoing indexing problem which gave rise to errors like "something horrible has happened in indexing".
The "best" log file for each process will now be recorded in a LogFiles directory - which means no more hunting around for the final scala log file and what not.
Chaling now produces unmerged reflection files in scalepack format as well as merged scalepack and merged MTZ format. This is done by scala (if used) by recycling the final SCALES and adding "output polish unmerged."
Now runs a twinning test using the CCP4 program sfcheck, and will warn you if your data look twinned.
Now allow up to 100 cycles of scale refinement to cope with more extreme cases where data at very different resolutions are scaled together.

Changes since 0.2.3

Added preferences file
Added xia2setup program to create the .xinfo file - this will also run LABELIT to configure the beam centre if it is installed
Added a strategy-of-data-reduction step to the pipeline
Added run-time check that CCP4 and so on are available

Changes since 0.2.2.4:

Changed to a BSD license
Added "python setup" check to xia2
Relaxed criteria on isomorphism to 1% not 0.5A etc.
Now wavelengths in wavelength record will override header values if provided, but do not (indeed, should not) be included if wavelength values are correct.

Changes since 0.2.2.3:

Created version for Power PC G4/G5
Fixed .csh setup scripts
If wavelength values not specified in the .xinfo file will use values from the image headers
If loggraph output from scala is broken by having Mn(I/sigma) greater than 99.99 can now cope (ignores that record)
Will now use a maximum of 180 degrees of data for deciding e.g. the point group, spacegroup and correct indexing standards - this helps the cases where exceptionally high redundancy data has been measured

Changes since 0.2.2.2:

Updated printheader to cope with MAR 165
Cleaner error messages when data missing
Added support for MAR image plate data
Included more example .xinfo files in the examples directory
Relaxed refinement parameters in Mosflm cell refinement

Platform Specific Advice

Gfortran compiled CCP4

Recent versions of the gcc compiler suite have included a new compiler called "gfortran" in place of "g77". This has, unfortunately, been designed with buffering too high on the agenda, and the output from programs can sometimes be mashed. If this happens you need to set "GFORTRAN_UNBUFFERED_ALL" to 1 in your environment.

Linux

There should be no particular problems on this platform - assuming that you have a recent version of Python. Any 2.4 version should work, but I have mostly used 2.4.3 and 2.4.4, and would recommend them. If the version you have on your system by default (type "python" to find out what version you have)is too old, you can easily install a new version, perhaps in the xia directory, from here. This is best installed by unpacking the tarball (tar xvfz Python-2.4.4.tgz) then doing ./configure /where/I/want/it then make, make install.

OS X.4 - X.6

The only potential problem with this platform is that the Python which comes as standard is 2.3 - however since you need cctbx.python anyway the problem has gone away.

Windows XP

Once more you will need a recent version of Python - which can be found here, however, you will also need "pywin32" to provide a little more job control functionality - this is available from here. Both are simple, double-click, binary installers.

There is now a version of Labelit available for Windows, which can be found here.

The mosflm binary which comes with iMosflm works well with xia2.