Data reduction

The LSST stack can be used to perform data reduction. It is not within the scope of this documentation to provide a tutorial for the LSST stack because one already exists here. However, the DRP can be used to setup a new Butler Repository and automatically ingest the science and master calibration files required by the user.

The automated approach

A ready-made Butler repository can be constructed using huntsman.drp.reduction.lsst.LsstDataReduction and subclasses. This includes ingestion of the science files, master calibration files as well as the catalogue of reference objects for photometry / astrometry.

The basic usage looks like this:

from huntsman.drp.reduction.lsst import LsstDataReduction as Reduction

query = {"observation_type": "science", "field": {"in": ["CenA, CentaurusA"]}}

reduction = Reduction(name="test_reduction",
                      query=query)

reduction.prepare()

This will create the Butler Repository in the “test_reduction” reduction of the “reductions” directory specified in the config.

Alternatively, we can create reduction instances from a yaml config file:

from huntsman.drp.reduction import create_from_file

reduction = create_from_file(reduction_filename)

Please see the Reduction API for further details.

The manual approach

First, create a Butler Repository instance:

from huntsman.drp.lsst.bulter import ButlerRepository

repo = ButlerRepository(directory_name)

The directory name can be any valid directory name. Typically this should be within the pre-existing “reductions” directory, which is mounted into the Docker containers.

One can then query for the science files they want to process and ingest them into the repo:

from huntsman.drp.collection import ExposureCollection

collection = ExposureCollection.from_config()

docs = collection.find({"observation_type": "science", "field": {"in": ["CenA, CentaurusA"]}})

repo.ingest_raw_files([d["filename"] for d in docs])

We can automatically find the best set of calibs for these science files and ingest them:

from huntsman.drp.collection import CalibCollection

calib_collection = CalibCollection.from_config()

calib_docs = defaultdict(set)
for doc in docs:
    cdocs = calib_collection.get_matching_calibs(doc)
    calib_docs.update(cdocs.values())

repo.ingest_calib_docs(calib_docs)

Now use the reference catalogue client to download reference sources and ingest them into the repo:

from huntsman.drp.refcat import RefcatClient

refcat_client = RefcatClient.from_config()

refcat_client.make_from_documents(docs, filename=refcat_filename)

self.butler_repo.ingest_reference_catalogue([self._refcat_filename])

Once the files are ingested into the repository, one can do the remainder of the processing using the LSST stack directly. There are also several methods in the ButlerRepository class that may be used to process the data, e.g. construct_skymap and construct_calexps.