This documents the lasting problems and how to go about fixing them

The I:NUTARZ device fell out in August for about ~0.2% of the spills. Currently this device is used to determine the location of the target for the purpose of projecting BPMs at 121 and TGT to the target and for determining what the beam energy (LE, psME, etc) is.

The fix:

1. Modify BMSpillFiller to not rely on NUTARZ directly
2. Establish a new table, say, BeamMonConditions, that holds the target location
3. Devise a mechanism to reliably update this table with the target location does change (eg, rely on human notification (yeah, right) or write a process that monitors NUTARZ and alerts experts when it changes).
4. Modify BDSpillAccessor to look at this table for beam energy, and modify BMSpillFiller to use it when projecting BPMs (or make this a post-DB calculation)
5. Refill DB

In the near term, it may be easier to just hard code the time periods of different NUTARZ values in BDSpillAccessor. Although, this won't fix the missing BPM values.

The bdp-server can go down. This has happened for these reasons:

• Unnecessary reboot of minos-acnet by clueless shifter
• Dead ethernet jack in CR
• The crash-after-long-beam-off problem. It is currently unknown as to the cause but this only happens when the beam turns after an extended down time (but not after every down time)

The primary fix is to run a second bdp-server (currently running under user "bv" on minos05). But, this needs better monitoring and needs to run under a group user account. This new backup process needs to be setup and plugged into shift monitoring.

There are various calibrations:

• TORTGT: early running has this ~5% miss-calibrated. This calibration was put in end of July 05. It required all processes running ACNET to restart. This requirement was not communicated so there is ~1 week of ~13% miss-calibration followed by a residual 0.6%. These are the actual values from Mary. They show the scaling to be applied to raw TORTGT values and the UTC Unix timestamped ranges:

  -4.8%

  before 1122344261

  -13.2%

  1122344261 to 1122936203

  -0.6%

  after 1122936203

• Horn currents: This has some offset. "185" kA is more like 182 and is measured at ~179. Maybe there is also a drift in time? Need exact values.
• BPMs: have electronic (ADC), electrical (cables) and physical alignments. Need corrections and dates.
• Hadron/muon monitors - calibration code in place, but needs checking when conditions change.

The fix for TORTGT, horn and BPMs is to put in yet another table. I think this should be a generic linear (quadratic) correction that could be used to store such for any device.

Currently the target is not at (0,0) and is in different locations as measured in the BPM and the PM coordinates. This offset may change in time if it is indeed due to floor sag.

This can be fixed by treating the alignments as an entry in the above mentioned calibration table.

Currently the beam widths are the statistical RMS of the PM at TGT. Mary has put in the code to instead do a Gaussian fit and store the sigma, including hot channel removal. This needs to be put in place by re-filling the DB. At the same time, Mark's BMSpillAna class needs a re-write so to change the beam width cut.

Mark found 6680 ND spills with no beam monitoring within 1 second. This is Sept 14-19 05. This problem is not yet understood but may be related to high load on the system running the bdp-server.

In addition to missing spills, toroids, NUTARZ and other devices go missing from spills. The source of thsi is not yet understood.

We need to (re)write the shift manual section on beam monitoring. In addition we need to produce some validation code that runs several times a day. This has its own page here.