I want to start this issue with a failure that can look like progress. A team trains a risk model, the offline numbers look strong, and the AI workflow around it feels more credible because there is finally a score beside the generated explanation. Then production behaves differently. The score is weaker than expected because the training data quietly used feature values that did not exist at the time of the original decision.

That is not a model problem first. It is a time boundary problem. Hybrid AI systems often combine classic ML scores, deterministic policy, retrieval, and LLM-generated explanations. If the features feeding that system are stale, missing, or leaked from the future, the rest of the architecture is already standing on unstable ground.

In this issue, we build an actual local Feast feature store. Feast reads a local Parquet offline store, registers a feature view and feature service, builds point-in-time training rows with historical retrieval, materializes features into SQLite, and serves online feature vectors for deterministic scoring.

The Boundary I Care About Here

A feature store is not just a convenient place to keep features. The serious production boundary is time. Training needs features as they existed at decision time. Online serving needs features fresh enough for the current request. Those are different contracts, and both can fail quietly.

Feast's feature retrieval docs give us the right vocabulary: historical retrieval through get_historical_features, online retrieval through get_online_features, feature services as named contracts, and event timestamps for point-in-time correctness. Feast's local provider uses a file offline store and SQLite online store by default, which is exactly the shape we use here.

That is the important move in this issue. We are not talking about feature stores abstractly. We are using Feast locally, then putting deterministic application policy around the feature data it returns.

What We Are Actually Building

The companion repository contains a Python project named point-in-time-features. It installs Feast, generates a local Parquet file from checked-in JSON fixtures, applies Feast feature definitions, materializes into Feast's SQLite online store, and scores driver risk requests from Feast online retrieval.

The app does this:

• loads a feature-service policy
• writes local Feast Parquet data
• applies Feast entity, source, feature view, and feature service definitions
• calls Feast get_historical_features for training retrieval
• reports future feature observations that were excluded
• turns missing Feast historical rows into rejected training cases
• calls Feast materialize_incremental into SQLite
• calls Feast get_online_features for runtime requests
• blocks online requests when Feast cannot return a fresh complete vector

The example uses driver risk scoring, but the real lesson is broader: production AI systems need to prove which feature values were available at the moment a model or agent made a decision.

The Shape Of The Feature Store

There are two paths in the system. The historical path creates training rows. The online path serves current decisions. Both go through Feast, and both share the same feature service.

Feast owns historical retrieval, materialization, and online retrieval. The application owns product policy around the returned vector: missing features fail closed, stale features fail closed, and the deterministic scorer only sees the active feature-service fields.

Local Feast Configuration

The actual Feast repository lives under feast_repo/. The configuration uses the local provider, a file offline store, a local registry, and SQLite online storage.

project: point_in_time_driver_risk
registry: data/registry.db
provider: local
offline_store:
type: file
online_store:
type: sqlite
path: data/online_store.db
entity_key_serialization_version: 3

That means the workflow is local but still real Feast. The generated Parquet file becomes the offline store. The registry and online store are local files. No cloud feature platform is involved.

The Feature Service Is The Contract

The feature definitions file declares the Feast entity, source, feature view, and feature service.

driver = Entity(
  name="driver",
  join_keys=["driver_id"],
  value_type=ValueType.STRING,
  description="Driver entity for local point-in-time risk decisions.",
  owner="platform-ai",
)

driver_activity_source = FileSource(
  name="driver_activity_source",
  path="data/driver_feature_observations.parquet",
  timestamp_field="event_timestamp",
  created_timestamp_column="created",
)

driver_activity = FeatureView(
  name="driver_activity",
  entities=[driver],
  ttl=timedelta(days=1),
  schema=[
      Field(name="completed_trips_30d", dtype=Int64),
      Field(name="cancellation_rate_30d", dtype=Float32),
      Field(name="avg_rating_30d", dtype=Float32),
      Field(name="late_arrivals_7d", dtype=Int64),
      Field(name="support_tickets_30d", dtype=Int64),
      Field(name="feature_event_unix_seconds", dtype=Int64),
      Field(name="source_event_id", dtype=String),
  ],
  online=True,
  source=driver_activity_source,
)

driver_risk_v1 = FeatureService(
  name="driver_risk_v1",
  features=[driver_activity],
)

I like this shape because the contract is boring and reviewable. It says which entity key joins features, which feature view owns the data, which TTL bounds historical lookup, and which feature service the application uses for both training and serving.

The application policy file mirrors that contract with score features and a freshness metadata field. Feast returns the vector. The app decides whether that vector is complete and fresh enough for the product decision.

Training Rows Come From Feast

Historical training rows are built through Feast, not a custom joiner. The pipeline creates an entity dataframe with driver_id, event_timestamp, case_id, and label, then asks Feast for the active feature service.

feature_service = store.get_feature_service(feature_service_policy.name)
historical = store.get_historical_features(
  entity_df=entity_df,
  features=feature_service,
  full_feature_names=False,
).to_df()

That call is the key boundary. Feast uses the entity key and event timestamp to return feature values as of the historical decision time. The one-day TTL on the feature view prevents old snapshots from being treated as valid training data.

The repo still records future observations as warnings. That does not mean Feast used them. It means the dataset had later values available and the run can prove they were excluded from the training row.

Missing Historical Rows Become Explicit Rejections

In the tested Feast version, historical retrieval returns the rows with eligible feature snapshots. Cases with no eligible snapshot inside the TTL do not become useful training rows. The application turns those missing Feast rows into explicit rejected cases.

for case in cases:
  if case.case_id in returned_case_ids:
      continue

  rows.append(
      TrainingRow(
          case_id=case.case_id,
          entity_id=case.entity_id,
          decision_at_utc=case.decision_at_utc,
          label=case.label,
          feature_service_name=feature_service_policy.name,
          feature_service_version=feature_service_policy.version,
          source_event_id=None,
          feature_observed_at_utc=None,
          feature_age_minutes=None,
          features=None,
          decision="reject",
          reasons=["no_eligible_feast_feature_snapshot"],
          warnings=warnings,
      )
  )

That is the behavior I want in production. A rejected training case is better than a quiet hole in the dataset. It gives the team something concrete to inspect: missing history, stale features, bad TTL, delayed ingestion, or a source-system problem.

Materialization Separates Offline History From Online Serving

Training retrieval and online retrieval are not the same operation. Historical retrieval builds many rows across many decision times. Online retrieval needs the latest materialized feature vector for one entity now.

The pipeline materializes through Feast:

with redirect_stdout(StringIO()), redirect_stderr(StringIO()):
  store.materialize_incremental(as_of_utc)

The output store is Feast's SQLite online store at feast_repo/data/online_store.db. In a real deployment, this could become Redis, DynamoDB, Datastore, or another online store. The contract around freshness and missing values should remain the same.

Online Decisions Read From Feast

Runtime scoring does not reach back into the JSON fixture or the Parquet file. It asks Feast for online features from the active feature service.

response = store.get_online_features(
  features=feature_service,
  entity_rows=[{policy.join_key: request.entity_id}],
  full_feature_names=False,
).to_dict()

row = {key: values[0] for key, values in response.items()}

If Feast returns missing values, the runtime blocks. If the retrieved feature timestamp is too old for the request time, the runtime blocks. Only a complete fresh vector reaches the scorer.

That separation matters. Feast is the retrieval system. Product policy still decides whether the retrieved vector is safe to use for this decision.

The Scorer Is Boring On Purpose

The sample scorer is deterministic. It turns a small set of operational signals into approve, watch, or manual_review.

if score >= self._policy.manual_review_threshold:
  decision = "manual_review"
elif score >= self._policy.watch_threshold:
  decision = "watch"
else:
  decision = "approve"

A real system could replace that scorer with logistic regression, gradient boosting, ML.NET, scikit-learn, XGBoost, or a managed model endpoint. The Feast boundary should not change. The scoring layer should still receive a versioned feature vector from the active feature service.

If an LLM enters this workflow, I would put it after the score. It can draft a human-readable explanation or summarize supporting evidence. It should not decide whether stale Feast features are acceptable. It should not invent missing feature values. It should not quietly change the feature service.

A Local Feast Run Tells The Story

Install the local environment first:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .
python run.py

The output is deliberately plain:

Point-in-time feature store
Feature service: driver_risk_v1 2026.09.12
Training cases: 5
Training rows admitted: 3
Training rows rejected: 2
Future observation cases: 5
Online entities materialized: 3
Online entities skipped: 2

REQ-001 | MANUAL_REVIEW | entity=driver-101 | score=0.90 | reasons=high_cancellation_rate,low_rating,frequent_late_arrivals,recent_support_ticket
REQ-002 | APPROVE | entity=driver-102 | score=0.00 | reasons=no_risk_signals
REQ-003 | WATCH | entity=driver-103 | score=0.49 | reasons=elevated_cancellation_rate,rating_needs_attention,recent_late_arrival,recent_support_ticket
REQ-004 | BLOCK | entity=driver-104 | score=none | reasons=missing_feature:completed_trips_30d,missing_feature:cancellation_rate_30d,missing_feature:avg_rating_30d,missing_feature:late_arrivals_7d,missing_feature:support_tickets_30d,missing_feature:feature_event_unix_seconds,missing_feature:source_event_id

Report: data\reports\point-in-time-report.json
Training rows: data\reports\training-rows.jsonl
Online store: feast_repo\data\online_store.db
Audit events: data\reports\audit-events.jsonl

Those counts are the architecture. Three historical cases become training rows through Feast historical retrieval. Two cases are rejected because Feast cannot provide an eligible point-in-time feature snapshot. Five cases had later feature observations available in the dataset, and the report records that those values were not used. Three entities make it into the online serving path. Two are skipped because their latest source features are stale for the materialization window.

The fourth online request is the runtime behavior I care about most. Feast returns no usable online vector for driver-104, so the runtime blocks. It does not get a weak score from stale data. That gives the surrounding AI system an honest state to handle: ask for fresh features, send the case to manual review, or delay the workflow.

The Tests Protect The Feature Boundary

The tests focus on the Feast boundary rather than model accuracy.

python -m unittest discover -s tests

The current suite passes 6 tests covering:

• Feast-backed pipeline output files and counts
• Feast historical retrieval excluding a future snapshot
• missing Feast historical rows becoming rejected training cases
• Feast online retrieval blocking missing feature vectors
• request-time freshness checks for materialized online features
• deterministic scoring thresholds

The Deterministic Boundary

The probabilistic layer may eventually summarize the score, draft a reviewer note, explain the top signals, or combine this risk result with retrieved policy. It does not own the feature boundary.

In this implementation, Feast owns:

• the feature registry
• the file offline store contract
• the feature view
• the feature service
• historical retrieval
• materialization
• SQLite online serving
• online feature retrieval

The application owns:

• which Feast feature service is active
• how missing Feast rows become explicit rejected cases
• future-observation reporting
• request-time freshness checks
• missing-feature blocking
• risk-score thresholds
• report and audit persistence

The model can still be useful. It just cannot rewrite time.

Why This Architecture Works

The value is that feature data becomes an explicit system contract instead of an invisible assumption behind model output.

• Training rows are generated by Feast historical retrieval.
• Event timestamps make point-in-time joins explicit.
• The feature view TTL bounds how far back historical retrieval can look.
• Future observations are visible as warnings instead of hidden leakage.
• Online decisions read from Feast's materialized SQLite store.
• The active feature service carries a name and version.
• The scorer receives a bounded feature vector, not arbitrary raw data.
• The tests protect the boundary that model evaluation would otherwise take for granted.

This is why feature stores matter in AI engineering. They are not only ML infrastructure. They are one of the places where data time, model behavior, and product decisions meet.

Potential Enhancements

The next version could add delayed labels, calibration reports, feature drift checks, shadow scoring for a new feature-service version, OpenLineage events for materialization, and OpenTelemetry metrics for online feature freshness and decision blocking.

For a larger production system, I would separate feature ownership by source team, add data-quality expectations before materialization, and require promotion gates before a new feature service becomes active. A feature change can be just as risky as a model change when the score feeds an AI workflow.

I would also add a second online store profile. SQLite is perfect for the local issue. Redis or another low-latency online store would make the serving tradeoffs more realistic once the feature boundary itself is understood.

Final Notes

A hybrid AI decision system can look sophisticated while still failing at a very old problem: using the wrong data at the wrong time.

The practical lesson is simple: make feature time explicit. Use Feast historical retrieval for training rows. Use Feast materialization and online retrieval for runtime decisions. Block when the feature contract cannot be satisfied. Then let ML models and LLMs operate inside that boundary, not around it.

Explore the companion repository at the GitHub repository.

See you in the next issue.

Stay curious.