SAGUARO TOM

Welcome to the SAGUARO target and observation manager for GW follow-up.

Repositories of Interest:

• https://github.com/SAGUARO-MMA/saguaro_tom.git
• https://github.com/SAGUARO-MMA/sassy_q3c_models.git
• https://github.com/SAGUARO-MMA/kne-cand-vetting.git

Installation (for development)

Prerequisites: Python 3.11 or 3.12 (django-autocomplete-light==4.0.0 requires 3.11+, and the tom-nonlocalizedevents dependency does not support 3.13+). A PostgreSQL database is required for the application.

1. Clone the repository:

  % cd /var/www
  % git clone https://github.com/SAGUARO-MMA/saguaro_tom.git

2. Copy settings_local.template.py to settings_local.py and edit as needed. At minimum, set SECRET_KEY, and the POSTGRES_* database settings so the app can connect. For local development you may set DEBUG = True.

  % cd /var/www/saguaro_tom/saguaro_tom
  % cp settings_local.template.py settings_local.py
  % vi settings_local.py

3. Create virtual environment and install dependencies.

Option A — Conda (from the project root, so requirements.txt is found):

  % cd /var/www/saguaro_tom
  % conda env create -f environment.yml
  % conda activate trove

Option B — venv + pip:

  % cd /var/www/saguaro_tom
  % python3 -m venv venv
  % source venv/bin/activate
  % pip install --upgrade pip
  % pip install -r requirements.txt

4. Apply migrations. Use PostgreSQL for the database; the project's migrations are not fully compatible with SQLite (e.g. tom_nonlocalizedevents.0016 can raise "near None: syntax error" on SQLite). On a fresh database, the packaged migration tom_targets.0025 (RunPython) can run before guardian or trove_targets exist in the migration state, and faking it can trigger post_migrate before tom_common is applied (causing "no such table: tom_common_profile"). So run a full migrate first (it will fail on 0025), then fake 0025, then migrate again:

  % python manage.py migrate
  % python manage.py migrate tom_targets 0025 --fake
  % python manage.py migrate

The first run applies everything up to and including tom_targets.0024 (and tom_common, guardian, etc.); it will then fail on 0025. Faking 0025 and re-running migrate applies the rest. On a new DB there are no "Public" permissions to migrate anyway, so faking 0025 is safe.

If you use SQLite, run python manage.py repair_migrate instead of migrate alone. It creates missing tables (e.g. tom_nonlocalizedevents_superevent), fakes renames when the target table already exists, and retries migrate by faking any migration that fails with "table already exists" or "no such column" until migrate succeeds. For one-off fixes you can still use python manage.py repair_superevent_table then migrate. Prefer PostgreSQL to avoid these issues.

5. Run the development server. The first time, Django may attempt to download the SFD dust map (network access required).

  % python3 manage.py collectstatic # only the first time you start the development server
  % python3 manage.py runserver

Then open http://127.0.0.1:8000/ in your browser to view the TOM. If you use PostgreSQL, create the database first (e.g. createdb trove) and set the POSTGRES_* values in settings_local.py to match.

Enabling WSGI under Apache2

Enable WSGI within Apache2 in the usual way (there are plenty of web tutorials on this).

Edit saguaro_tom.conf and saguaro_tom-ssl.conf as you see fit to reflect your source code directory and security certificates. You should probably rename 'localhost' to something more suitable. If you do, you will also need to add that name to /etc/hosts and set ALLOWED_HOST in settings_local.py. If you are hosting from a subdomain, you must also set FORCE_SCRIPT_NAME in settings_local.py.

Then (as root) enable the sites:

  % cd /var/www/saguaro_tom
  % cp saguaro_tom.conf /etc/apache2/sites-available/
  % cp saguaro_tom-ssl.conf /etc/apache2/sites-available/
  % cd /etc/apache2/sites-available/
  % apache2ctl configtest
  % a2ensite saguaro_tom.conf
  % a2ensite saguaro_tom-ssl.conf
  % service apache2 restart

Running the alert listener

The alert listener is now integrated into the TOM. It should automatically restart when sand restarts, thanks to this cronjob (run as root):

@reboot sleep 40 && /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py readstreams > /home/saguaro/alertstreams.log 2>&1

If it does not restart, or you need to restart the listener manually, run the following on sand. First, kill any other instances are running:

pkill -f readstreams

Then run

nohup /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py readstreams > /home/saguaro/alertstreams.log 2>&1 &

Allowing for asynchronous tasks

To run asynchronous tasks (ATLAS forced photometry queries and minor planet checking), we use django-tasks, configured according to the TOM Toolkit documentation. The asynchronous workers should automatically restart when sand restarts, thanks to this cronjob:

@reboot sleep 60 && cd /var/www/saguaro_tom/ && for i in $(seq 1 48); do venv/bin/python manage.py db_worker --queue-name '*' -v3 > /home/saguaro/django_tasks_logs/django-tasks.${i}.log 2>&1; done

If the workers do not restart, or you need to restart them manually, run the following on sand:

pkill -f db_worker
cd /var/www/saguaro_tom/
source venv/bin/activate
for i in $(seq 1 48)
    do nohup venv/bin/python manage.py db_worker --queue-name '*' -v3 > /home/saguaro/django_tasks_logs/django-tasks.${i}.log 2>&1 &
done

To view what their logs all at once, run:

tail -f ~/django_tasks_logs/*

Press Ctrl+C to exit that view.

Other periodic tasks

Several other tasks run every hour as cronjobs (as root):

0 * * * * /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py report_pointings > /home/saguaro/report_pointings.log 2>&1
0 * * * * /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py updatestatus > /home/saguaro/observation_status.log 2>&1
0 * * * * /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py verify_listener > /home/saguaro/verify_listener.log 2>&1
5 * * * * /var/www/saguaro_tom/venv/bin/python /var/www/saguaro_tom/manage.py ingest_tns > /home/saguaro/ingest_tns.log 2>&1

Respectively, these:

• report survey pointings to the Treasure Map
• update the statuses of observations scheduled through the TOM (e.g., MMT)
• verify that the GW alert listener is functioning by checking that we received an alert for the latest event in GraceDB
• ingest targets from the TNS (the TNS table is updated on the hour, so we run this 10 minutes after)