Merge trigger files
In order to not end up with millions of small .root, .hdf5, and
.xml files each representing a
small chunk of processed time, the omicron-process workflow will merge
contiguous files together using the omicron-merge-with-gaps command line utility.
The purpose of this control utility is to detect any gaps in the expected list
trigger files. The gaps are data dependent failures of Omicron to process
one of the short time intervals.
omicron-merge-with-gaps uses the following command-line utilities to
merge te contiguous trigger files:
File type |
Extension |
Program |
|---|---|---|
Root |
|
|
HDF5 |
|
|
ligolw |
``.xml `` |
|
Text |
``.txt `` |
|
The omicron-root-merge executable is a thin wrapper on top of
the omicron.io.merge_root_files() method:
The omicron-hdf5-merge executable is a thin wrapper on top of
the omicron.io.merge_hdf5_files() method:
The ligolw_add is an external program contained in the lscsoft-glue package.
Command-line options
For detailed documentation of all command-line options and arguments, print the --help
message of each program:
$ omicron-merge-with-gaps --help
/home/docs/checkouts/readthedocs.org/user_builds/pyomicron/conda/latest/lib/python3.11/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems
_warn(("h5py is running against HDF5 {0} when it was built against {1}, "
usage: omicron_merge_with_gaps [-h] [-v] [-V] [-q] [-l LOG_FILE] [-o OUT_DIR]
[-n] [--no-gzip] [--uint-bug]
[--file-list FILE_LIST]
[infiles ...]
Handle merging different trigger file types detecting any gaps
positional arguments:
infiles List of paths to files to merge or copy
options:
-h, --help show this help message and exit
-v, --verbose increase verbose output
-V, --version show program's version number and exit
-q, --quiet show only fatal errors
-l LOG_FILE, --log-file LOG_FILE
Save log messages to this file
-o OUT_DIR, --out-dir OUT_DIR
Path to output directory for merged files
-n, --no-merge Do not merge files, only copy to output indir
--no-gzip Do not compress the ligolw xml files
--uint-bug Fix problem XML files created by old version of
Omicron before merging.
--file-list FILE_LIST
File with list of input file paths, one per line
$ omicron-root-merge --help
/home/docs/checkouts/readthedocs.org/user_builds/pyomicron/conda/latest/lib/python3.11/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems
_warn(("h5py is running against HDF5 {0} when it was built against {1}, "
usage: omicron-root-merge [-h] [-V] [-d] [-s] filename [filename ...] output
Merge multiple Omicron ROOT files into one
positional arguments:
filename file to merge
output output file name
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-d, --remove-input remove input files after writing output (default: False)
-s, --strict only merge contiguous data (default: False)
$ omicron-hdf5-merge --help
/home/docs/checkouts/readthedocs.org/user_builds/pyomicron/conda/latest/lib/python3.11/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems
_warn(("h5py is running against HDF5 {0} when it was built against {1}, "
usage: omicron-hdf5-merge [-h] [-V] [-d] filename [filename ...] output
Merge HDF5 files into one
positional arguments:
filename file to merge
output output file name
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-d, --remove-input remove input files after writing output (default: False)
$ ligolw_add --help
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/pyomicron/conda/latest/bin/ligolw_add", line 37, in <module>
from lal.utils.cache import CacheEntry
ModuleNotFoundError: No module named 'lal'
Reducing file count and disk space
Current operations run omicron-proocess every 12 minutes to provide near
real time triggers. Over a day this can result in up to 360 (5 * 24 * 3) files per
channel with O(1000) channels per day.
To address this issue, the program omicron-metric-day-merge will scan each channel
for trigger files in a specific metric day. A metric day is used to mean 100,000 second
boundary of GPS time. It will colaesce all contiguous trigger files into a temporary diretory
Archive trigger files
Archiving is also complicated by the potential gaps in triggers. The omicron-archive
program examines all the merged trigger files organizing them by channel and metric day
(int(gps_start_time/1e5)). These are then copied to the archive directory, usually
/home/detchar/triggers/<ifo>, but it can be overridden for testing on the command
line or by setting the envirnment variable OMICRON_HOME. The full help opions are:
$ omicron-merge-with-gaps --help
/home/docs/checkouts/readthedocs.org/user_builds/pyomicron/conda/latest/lib/python3.11/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems
_warn(("h5py is running against HDF5 {0} when it was built against {1}, "
usage: omicron_merge_with_gaps [-h] [-v] [-V] [-q] [-l LOG_FILE] [-o OUT_DIR]
[-n] [--no-gzip] [--uint-bug]
[--file-list FILE_LIST]
[infiles ...]
Handle merging different trigger file types detecting any gaps
positional arguments:
infiles List of paths to files to merge or copy
options:
-h, --help show this help message and exit
-v, --verbose increase verbose output
-V, --version show program's version number and exit
-q, --quiet show only fatal errors
-l LOG_FILE, --log-file LOG_FILE
Save log messages to this file
-o OUT_DIR, --out-dir OUT_DIR
Path to output directory for merged files
-n, --no-merge Do not merge files, only copy to output indir
--no-gzip Do not compress the ligolw xml files
--uint-bug Fix problem XML files created by old version of
Omicron before merging.
--file-list FILE_LIST
File with list of input file paths, one per line