Production Monitoring Engineer

The problem.

The system is unattended most of the time. That is safe exactly to the extent that abnormality is visible early. The failure that costs money is rarely a crash. It is a component that keeps running while its inputs stop meaning anything. Detecting that is harder than detecting death. That is what this role is for.

What you would do.

• Define and build health signals for every component, including ones that require knowing what the data should look like.
• Build the dashboards the desk actually watches. Delete the ones nobody does.
• Own the alert rules: what fires, at what threshold, to whom, and what the person is expected to do about it.
• Detect staleness, drift and silent degradation, not only errors and absence.
• Keep monitoring independent enough that it survives the failure it is meant to report.

What we would want evidence of.

• You have built monitoring that caught something before a person did, and can describe it.
• You have removed alerts and the on-call got better.
• You understand the difference between a system that is down and a system that is lying.
• You have made monitoring independent of the thing it monitors, and know why that matters.
• You design for the person reading the alert at three in the morning, including when it is you.

What would make this a bad fit.

This role is a poor fit if you want to install a platform and configure it. Most of the value here is in domain-specific checks that no platform ships with.

This role is a poor fit if dashboards are the deliverable. The deliverable here is a decision someone made in time.