Engineering tests 2018-Aug-17 and 18th (Billy Barott); summary of notes (see full obsnotes for detailed commands).

* Ran BF initialization and found that SPK daemons were down, which prevented init. On boot2, ran SPK daemon unload and reload commands. Cleared the error without incident.

* Ran the BF health check scripts from 2013; ran on BF1 and BF2, L and S band cals. Created lists of potentially-usable antennas. These antpols are:

* L-Band * [Bf1/ First run] 1ax,1ay,1bx,1by,1fx,1fy,1gy,1hx,2bx,2by,2dy,2ex,2ey,2fy,2jx,2jy,2kx,2lx,3ex,3ey,3jx,3jy,3lx,3ly,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5ey,5gx,5gy,5hx,5hy

[BF1/ Second run] 1ax,1ay,1bx,1by,1dx,1dy,1fx,1fy,1gy,1hx,2ax,2ay,2bx,2by,2dy,2ex,2ey,2fy,2jx,2jy,2kx,2lx,2my,3ex,3ey,3jy,3lx,3ly,4ex,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5ey,5gx,5gy,5hx,5hy

[BF2] 1ax,1ay,1bx,1by,1dx,1dy,1fx,1gy,1hx,2ax,2bx,2dx,2dy,2ex,2ey,2fy,2jy,2kx,2lx,2my,3ex,3ey,3jx,3jy,3lx,3ly,4ex,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5gx,5gy,5hx

* S-Band * [BF1] 1ax,1ay,1dy,1fx,1fy,1gy,1hx,2ax,2ay,2bx,2by,2dx,2dy,2ex,2ey,2fy,2jx,2jy,2kx,2my,3ex,3ey,3jy,3lx,3ly,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5ey,5gx,5gy,5hx,5hy

[BF2] 1ax,1ay,1dy,1fx,1gx,1gy,1hx,2ax,2bx,2dx,2dy,2ex,2ey,2fy,2jx,2jy,2kx,3ex,3ey,3jy,3lx,3ly,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5gx,5gy,5hx

Initialized all three beamformers with the antlist determined by the BF1 health check at L-band, and processed delay calibrations (30s on CASA).

Beamformers #2 and #3 displayed many link errors that were not cleared by firmware reinit or hardware reinit. Further attempt was to drop all antennas on related iBobs from the hookup (i9,16,22,24). This appears to have cleared the link errors on the main nodes; did not clear link alarms on B15.bfa (recently repaired unit), which shows ERCS alarms on three FPGAs. However, this might be sufficient for Foxtrot obs (doesn't use b15). After some time link error also showed on BF3 link to i21. Was not pursued further.

New antenna lists are:c (BF1) 1ax,1ay,1bx,1by,1dx,1dy,1fx,1fy,1gy,1hx,2ax,2ay,2bx,2by,2dy,2ex,2ey,2fy,2jx,2jy,2kx,2lx,2my,3ex,3ey,3jy,3lx,3ly,4ex,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cx,5cy,5ex,5ey,5gx,5gy,5hx,5hy

(BF2/3) 1ax,1ay,1bx,1by,1dx,1dy,1fx,1fy,1gy,1hx,2ax,2ay,2bx,2by,2ex,2jx,2jy,2kx,2lx,2my,3ex,3jy,3lx,3ly,4ex,4gx,4gy,4jx,4jy,4kx,4ky,4ly,5cy,5ex,5gy,5hx

#========== CASA Stability Took 30 snapshots each of crosses and outputs on CASA with all 3 beamformers. Crosses activities: 2018-8-18-8-41-4-BF1: 2018-8-18-8-41-5-BF3: 2018-8-18-8-41-7-BF2: Out activities are 2018-8-18-8-57-52-BF1 2018-8-18-8-56-34-BF2 2018-8-18-8-56-24-BF3

BF1: 5gx, 3ex,B2F4, showed significant drift over the integration. However, could be CASA. BF3: B1F3 very noisy. B2F4 some drift. B3F4 some digital - comb noise? B4F3 possible solar. Again, CASA? BF2: Similar

Autos on BF1 and BF2 showed the comb seen in previous data and attributed by suggestion of Jon to the new feeds, but are otherwise unremarkable. Autos on BF3 show very poor roll-off response and are indicative that BF3 is at present unusable.

Cross correlation results on 3c84, showing some solar interference, but otherwise maybe usable: 2018-8-18-9-32-44-BF1 2018-8-18-9-32-46-BF2

Back to CASA as a sanity check: 2018-8-18-9-53-47-BF1 2018-8-18-9-53-48-BF2

Have noticed in this and other runs that 5cx autos on casa in particular change amplitude significantly; network issue (soft start/stop, but ought be calibrated out with memtimers) or antenna issue?

Phase calibration was difficult. During phase cal gave BF2 to Janusz to do hunt for 8 MHz comb. Eventually tried 5 minute calibration on 3c48 while setting; useful, but not totally enough to completely beat down the solar interference. Final cal used +0.2 caloffset, but probably mistake at this tuning (1660).

Daytime calibration moving up to 6 GHz did show W3OH and g188.95m, but very weakly and didn't appear to be a good cal.