RELEASE_NOTES

# Copyright (c) 2013-2020, California Institute of Technology, all rights reserved

2020-Feb-19: v1.2.1 - Bug fix for Second Minor Release

- fixed a fatal bug when using BM twilight flats (skips ledge correction)

- fixed a fatal bug that set the wrong region in the geometry for masked data


2020-Feb-11: v1.2.0 - Second Minor release

- many minor bug fixes and plot and output improvements

- added ability to zoom line plots by dragging mouse in ktv.pro

- improved handling of twilight and dome flats

- fixed minor bug in bad column correction

- fixed major bug in handling nod-and-shuffle subtraction in stage1

- added capability to handle aborted nod-and-shuffle observations including
	the ability to apply a scale factor to the sky panel

- added /biasskip1 keyword to allow skipping of first bias in group

- added /skiposcansub to allow skipping of overscan subtraction step


2018-Jun-11: v1.1.0 - First Minor release

- use delta wavelength map to calculate more accurate effective area curves.

- added delta wavelength 2-D map to keep track of the native wavelength bins.

- added knotspp keyword to pipeline parameters to control the number of knots
	per Y pixel when using b-splines to fit sky (defaults to 1.25).

- fixed bug in variance calculation when sky model is used instead of sky obs.

- fixed bug in kcwi_masksky_ds9.py that swapped image dimensions.

- use OGIP method for putting long strings into keywords in FITS headers.

- added geometry file simulator for KCRM (in development).

- added user contributed software section to the AAAREADME file.


2018-Mar-12: v1.0.1 - First full release

- implemented saveplots ppar keyword that saves a few (2) or many (3) plots.

- implemented ghost busting in kcwi_trace_cbars.pro.

- added python utility for converting ds9 region files into a sky mask
	image.  Named kcwi_masksky_ds9.py and found in devel subdirectory.

- added scaling of non-local sky by exposure time.

- added tweaking of sky scaling enabled by setting display to 2 or more
	when running kcwi_stage5sky.pro.

- depricated kcwi_quick.pro to devel as normal routines can be used at
	telescope with no performance delay.

- various minor bug fixes and comment string and text output updates.


2018-Jan-08: v0.6.0 - Twentieth beta release

- added new tool: kctv.pro for display and analysis of KCWI 2 and 3d images.

- added coyote graphics library to support kctv.pro.

- now fix bad columns before CR cleaning, re-ordered, expanded mask values.

- now flag un-corrected pixels in master flat and flat corrected object images.

- major restructuring to implement the usage of bsplines to fit both the
	pixel-to-pixel and the illumination corrections and to fit and 
	subtract the sky (see AAAREADME for more details).

- added differential atmospheric refraction correction stage as stage 7 and
	renamed flux standard star correction stage to stage 8.

- fixed a bug in the coordinates for nod-and-shuffle images that was using
	the background coords instead of the object coords.

- added keyword FIRST to kcwi_prep to indicate first imgnum to process

- added interactivity to flux standard calibration to improve fits.

- masks now maintaining byte format and now include more values.

- changed median stack of cont flats to mean to account for intensity decay.

- changed CR rejection exposure time threshold from 60s to 2s.

- many output fixes and improvements to make solutions more robust.

- removed unused routines from 'devel' subdirectory.

- updated bad pixel lists.

- added scattered light subtraction to stage 2.

- added full configuration calibration report at the end of kcwi_prep.


2017-Jun-29: v0.5.0 - Nineteenth beta release

- removed link file paradigm for process control and replaced with a proc file 
	that uses keyword/value pairs for calibration associations.

- added kcwi_prep keyword ALTCALDIR for specifying alternate cal directory.

- renamed master bias, dark, flat files to contain originating image name.

- added optional psf model 'gaussy' for cr rejection that doesn't eat sky lines.

- general cleanup and reordering of log file outputs.


2017-May-19: v0.4.2 - Eighteenth beta release

- 'NASMASK' state no longer used to match configurations.

- match biases on 'gainmul' keyword.

- in data/stds added KECK formatted starlist kderp_stds_starlist.txt.

- made ThAr the reference for geometry on all configs until FeAr lamp replaced.

- show illumination type in image summary.


2017-May-12: v0.4.1 - Seventeenth beta release

- New non-iterative wavelength solution that works well for all gratings and
	slicers, old version available through keyword to kcwi_prep.

- Now uses asymmetric slice fit with higher order in y direction.

- Now outputs flattend cube in *_2d.fits for each arc as diagnostic.

- Writes out extracted arc spectra in *_arcs.fits file.


2017-Apr-21: v0.3.2 - Sixteenth beta release

- Many improvements to direct mode pipeline including more robust tracing and
	a new direct relative response stage (kcwi_stage6drr).

- Added bad column fixing for ALL, TBO ccd amp modes (more to come).

- Now trim output data cubes.

- Set exposure time threshhold for CR rejection of 60s.

- Tested WCS against real sky data and updated.

- Restructured data cube for more rational WCS application.

- Added Keck extinction curve to calibrations.

- Now kcwi_prep reports on uncalibrated geometry configurations.

- Improvements to target name keyword handling

- Adjusted output dispersions to be closer to native dispersions.

- Fixed bugs in kcwi_quick and tested on commish data.

- Now outputs a reverse geom image *_wavemap.fits to show wavelength mapping.


2017-Apr-06: v0.3.1 - Fifteenth beta release

- Improve calibration file grouping and associations.

- Implemented initial version of direct imaging pipeline.

- Now accommodates larger image sequence numbers.

- Handles dome flats at Keck correctly.

- Correctly extracts image file root and sequence number given Keck protocol.

- Added devel routine to fix header keywords (kcwi_fix_headers.pro).

- First pass at sky geometry keywords.


2016-Oct-07: v0.2.13 - Fourteenth beta release

- Improved dark image grouping and association and now use TELAPSE instead 
	of XPOSURE for matching criterion.

- Update keywords to XPOSURE and FRAMENO.

- Removal of CWI flexure routines.

- Added timing log output for solving geometry alone.

- Added logging of slice geometry solution residuals.

- Make arc-atlas match criterion based on atlas resolution element.

- Handle case where no lines are matched for a given bar in kcwi_plot_arcfits.

- Modified and tested nod-and-shuffle implementation.

- Added starting inter-bar cross-correlation positions for all gratings.

- Changed criterion for picking overscan fit order to be based on
	number of amps.

- Put in a check for monotonicity prior to call to cspline, which requires it.

- Check both nod-and-shuffle mask name and position number to assert status.

- Updated configuration matching criteria to allow for small variations in
	floating point values.

- Implement preference for ThAr arc for BH gratings.

- Changed coord check for cal images from warning to info.


2016-Aug-29: v0.2.12 - Thirteenth beta release

- Changed default image prefix to "kcwi"

- Added image rectification (based on CCD amp mode) segment to stage1.

- Fixed bug in geometry grouping.

- Changed overscan calculation to use lower order (5 to 2), to skip first 50 
	read pixels.

- Removed CWI_DATA declaration from startup.pro

- Now writes out atlas wavelengths used in full-CCD solution.


2016-Jul-12: v0.2.11 - Twelfth beta release

- Major work on arc line finding to be more robust for all KCWI modes.

- Removed RCS "Id" keywords from source code.

- Removed code specific to Palomar CWI.

- Adapted code to handle KCWI FITS header keywords.

- Made adjustments to allow solution of geometry using BH3 grating and binning
	1x1 pixels.


2015-APR-27: v0.2.10 - Eleventh beta release

- Replaced clnpeaks routine with findpeaks that produces a more robust list
	of significant, isolated lines in kcwi_solve_arcs.pro.

- Added cosine bell taper to spectra prior to cross-correlations to minimize
	edge effects in kcwi_fit_center.pro.

- Added taperfrac as a wavelength fitting parameter to allow user to handle
	severe edge problems caused by very bright arc lines near edges.

- Removed obsolete wavelength fitting parameters pkiso and pksig, and
	implemented pkdel as fraction of spectral resolution in Angstroms.

- Improved robustness of central wavelength calculation for CWI MEDREZ grating.

- Refined wavelength solution diagnostic plots.


2015-FEB-27: v0.2.9 - Tenth beta release

- Performing the profile correction in stage 5 is now optional for input
	to stage 6 (relative response).

- Added a keyword switch to give dome flats priority over twilight flats,
	overriding the default behaviour which is to give the reverse priority.

- Added quick start instructions to AAAREADME.

- Now cwi_format_info.pro errors out if allhdr.txt not found.

- Copied legend.pro to kcwi_legend.pro to avoid conflict with IDL v8.x.


2015-JAN-16: v0.2.8 - Ninth beta release

- Now give priority to twilight flats for relative response correction.

- Now query user for good twilight flats in cwi_format_info.pro.

- Implemented IFU position, set to 1 (Large) for CWI data.

- Replace calls to mean and stddev with a single call to moment in order to
	accommodate IDL v8.4.

- Rename standard calibration files from *_std.fits to *_invsens.fits.

- Made kcwi_quick.pro more robust based on at-telescope testing.


2014-NOV-14: v0.2.7 - Eighth beta release

- Made standard star calibration more robust: smoothing kernel limits and
	comparing calibration and object wavelength scales before application.

- Added verbose and display keywords to each stage procedure allowing the user
	to override the values in the ppar file.

- Put calculation of geometry stats (rms) after all tweaking and cleaning
	of coefficients are completed.

- Now kcwi_prep automatically associates standard star observations as long
	as they are named like the reference files in !KCWI_STDS/stds

- Improved CR/ghost rejection in kcwi_fit_flat: reject on initial fit
	residuals with 5 sigma threshhold.

- Altered fit plotting method in kcwi_fit_flat: now can select column to plot.

- Changed sigma clip values for cosmic ray rejection: now uses different
	values for cflat images and for short exposures to avoid bad clipping.

- Added plot of spectral line residuals to diagnostic plots output during
	geometry/wavelength solution generation.

- Changed overscan subtraction to subtract fit of column-collapsed row vector 
	instead of row-by-row mean subtraction, added diagnostic plots.

- Changed /skip1bias keyword to /cwi which also implements associations that
	attempt to account for flexure.

- Improved diagnostic outputs to kcwi_stage6rr.pro.

- Fixed WCS keywords in relative response (stage6) and std response 
	(stage7) images.

- Fixed bug in variance calculations for stages 1, 2, and 6.

- Added update keywords to cwi_format_info.pro and cwi_fix_headers.pro to
	allow easier use for at-telescope reductions.

- Various minor bug fixes.


2014-OCT-07: v0.2.6 - Seventh beta release

- Changed rotation angle in WCS CD matrix calculation to be -ROTPA and
	verified orientations with cluster field (M34).

- Added ROTPA to KCWI_CFG struct and image summary printout.

- Added kcwi_quick.pro that allows for quick-look processing of a single
	object including the option to display the data cube in ds9.

- Added infrastructure to allow selected processing of images for each stage.

- Modularization of kcwi_stage4geom.pro and many changes to improve
	robustness of wavelength fits.

- kcwi_stage7std.pro now applies extinction correction to data before
	calculating response function.

- cwi_format_info.pro now queries user for IR filter image range.

- Calculates 'middle' wavelength; plots it along with calculated central
	wavelength and adds header keyword WAVEMID to data cubes.

- N&S wavelength solution now produces rms fit diagnostics.

- Now trims data cubes correctly according to wavelength solution.

- Made wavelength solutions robust for negative and/or larger dispersions.

- Calibrated CWI MEDREZ grating and populated variables in routines with
	measured values.

- Added diagnostic output for wavelength solutions obtained by setting
	ppar.display = 3.

- Added status of geometric/wavelength fit preventing cubes from being
	written when solution is bad: status number indicates failure mode.

- Prevent infinite loop in splinefit.pro (called by kcwi_fit_flat.pro) when
	CHISQR value is a NaN.

- Modified kcwi_fit_flat.pro to handle NaN's returned by splinefit.pro
	handle residual cosmic rays.

- Improved robustness of wavelength solution

- Added diagnostic output when configurations do not match up


2014-AUG-25: v0.1.5 - Sixth beta release

- Bug fix in kcwi_group_geom.pro where non-standard calibration sequences
	could cause the routine to crash.

- Calibrated CWI YELLOW grating and populated variables in routines with
	measured values.


2014-AUG-20: v0.1.4 - Fifth beta release

- Prints DRP version in all log files.

- Uses fiducial wavelength to set zeropoint allowing easier coaddition 
	of data cubes: (3000A)+N*dw (N-integer, dw-disp in A/px).

- Added variables for CWI YELLOW grating.

- Now populates grating tilt anomoly keyword to derive more robust
	preliminary estimates of the central dispersion.


2014-JUL-30: v0.1.3 - Fourth beta release

- No longer scale FITS images into two-byte integer pixels.

- Now scale the stepsize for tracing bars by binning.

- Fixed truncation of long integer numbers (int to longint).

- Now uses first image of group as master cal image number.

- Changed associations to be more sophistocated.


2014-JUN-04: v0.1.2 - Third beta release

- Changed handling of default directories and changed output default from 
	'Redux' to 'redux'.

- Derives image number digits automatically from fits files in input dir if
	not set with keyword parameter.

- Now takes image name prefix and the number of digits in the image number
	as keyword parameters to KCWI_PREP.


2014-MAY-23: v0.1.1 - Second beta release

- Handles nod-and-shuffle observations that were stopped before completion.

- Now correctly accounts for exposure times in standard star calibration.

- Adds calibration image numbers to output fits headers.


2014-APR-25: v0.1.0 - First beta release

- Added kcwi_stage7std.pro routine to use standard star observation to
	generate a calibration for object data cubes.

- Apply profile, relative resp. corrections to nod-and-shuffle sky and
	object image cubes.

- Create mask and variance images for all image types.

- Reworked linking and ppar paradigm to use one master for each: 
	kcwi.ppar, kcwi.link.

- Now write out master flats, profile, and relative response images as 
	floats to avoid dynamic range truncation.

- Added diagnostic plots to kcwi_fit_flat.pro, adjusted N&S fitting limits.

- Now processes all calibration frames through.

- Now checks initial reference bar for scattered light and subtracts if
	needed before cross-correlation to determine initial offset.

- Added option to find central cross-correlation peak when doing initial
	wavelength solution.

- Fixed typo in deriving isolated reference peaks.


2014-MAR-21: v0.0.7 - Seventh alpha release

- Added routine to clean errant bars in wavelength solution coefficients.

- Updated KCWI_PPAR struct definition to include switch for cleaning
	coefficients: CLEANCOEFFS, defaults to 1 - "ON", but can turn off 
	by setting to 0.

- Added flag in KCWI_PREP to turn off coefficient cleaning: /NOCLEANCOEFFS.

- Made wavelength solutions more robust by adjusting line-matching threshhold
	when no matches are found regardless of iteration number (previously
	only adjust on first iteration only).

- Handle failure to trace continuum bars and generate geometry.

- Account for scattered light background in continuum bars images.


2013-DEC-20: v0.0.6 - Sixth alpha release

- Wavelength solution now uses cross-correlation and is automated (thanks to MM!).

- DISPLAY ppar variable now controls level of interaction: values above 1 are
	interactive, values of 1 show plots, but don't require input (batch),
	values of 0 don't show any plots.

- Updated cwi_format_info.pro to handle dflat (dome flat) image type.


2013-NOV-22: v0.0.5 - Fifth alpha release

- Data cubes now have 3-D WCS that is readable by ds9 and QFitsView.

- Changed data cube orientation to accommodate WCS: spatial axis is now x:RA,
	slice axis is now y:DEC, wavelength axis is still z:LAMBDA.

- Change profile image orientation to accommodate data cube re-orientation.

- Added kcwi_stage6rr, which accounts for slice-to-slice response variation.

- Many logging and plotting changes and enhancements.


2013-OCT-01: v0.0.4 - Fourth alpha release

- Added nod-and-shuffle region keywords and made them 1-biased in header,
	but 0-biased in the pipeline.

- Updated 'allhdr.txt' and 'hdrfix_example.txt' files to include the
	nod-and-shuffle region keywords.

- Continued to experiment with WCS keywords using testwcs.pro.


2013-SEP-26: v0.0.3 - Third alpha release

- Added slice profile correction and crude WCS to profile images in stage 5.

- Uses the reference slice (defined by setting the reference bar) to determine
	the average bar spacing and enforce this on all slices.

- Added pixel-to-pixel response function correction using continuum flats in
	stage 3.

- Added dark frame substraction stage in stage 2.

- Added verification routines and printing/logging routines.

- Allow grouping by object/configuration to allow use of same output wavelength
	solution.

- Can now set output wavelength parameters in ppar file to override defaults.

- Fixed a bug in the association of 'cbars' files to observations.

- Added padding at both ends of slices.


2013-AUG-08: v0.0.2 - Second alpha release

- Fixed associations to be based on configuration first, then on nearest in 
	time.

- Outputs image*_sky.fits and image*_scube.fits for nod-and-shuffle 
	observations.

- Uses a standard dispersion for each blue or red grating image that accounts
	for binning.

- Added new routine, cwi_format_info.pro, to use the *.info files that are 
	output for each CWI image for populating the image headers.

- Many tweaks of plotting and outputs.


2013-AUG-02: v0.0.1 - Initial alpha release

- This is based on work with CWI data and reduces both RED and BLUE 
	grating data.

- Handles Nod-and-shuffle sky subtraction.