Data Engineer, Market Data

The problem.

Every result the firm produces is bounded by the quality of what was recorded. A history assembled from what exists today has already been filtered for survivors. The filtering is invisible. The backtest looks fine. Keeping a record that does not lie about the past is the highest-leverage thing in the data layer.

What you would do.

• Own the capture: what is recorded, at what resolution, with what guarantee that a gap is visible rather than silently interpolated.
• Keep the universe record append-only. Every instrument that ever existed, with the date it appeared and the date it stopped, written once and never edited.
• Build the reconciliation that catches a defective backfill before it is used. We have shipped one that stamped the present onto the past.
• Design storage so a five-year query is routine and a point-in-time query is exact.
• Make the data contract explicit, so a researcher knows what a field means without asking.

What we would want evidence of.

• You have found survivorship bias in a dataset somebody trusted, and can describe how.
• You treat an append-only record as a design constraint rather than a preference.
• You have built a pipeline where a partial failure is detected rather than absorbed.
• You know what a point-in-time correct dataset costs to maintain, because you have maintained one.
• You are willing to make a query slower to make it honest.

What would make this a bad fit.

This role is a poor fit if you want scale as the challenge. Volumes here are moderate. The difficulty is correctness.

This role is a poor fit if you regard data quality as somebody else's acceptance criteria. This role is the person that somebody is.