· Updated · Abdulaziz Ayman Aldharrab
Building AUJ, our local-first smart retail shelf
Leading our Software Engineering capstone and building its computer-vision and demand-forecasting pipelines.
AUJ was our senior Software Engineering capstone project. I led the five-person team and handled the AI and machine-learning work. We built a proof-of-concept smart shelf that could monitor stock, verify shelf interactions, forecast demand, and show the results on a live dashboard.
The problem
Small retailers often depend on manual stock counts. That means a stockout can sit unnoticed, and the retailer gets very little information about what customers pick up, return, or leave behind.
Our prototype turned a conventional shelf into a local monitoring system. A load cell measured changes in weight. A camera captured a short burst when the reading stabilised. The shelf then sent the event to a local hub, which checked it, updated inventory, generated alerts, and pushed the result to the dashboard.
My role
As team lead, I was responsible for coordinating the project and for the machine-learning pipeline. My technical work covered two separate problems: using computer vision to verify what happened at the shelf, and forecasting demand for each product.
Creating and training the vision model
The computer-vision model did not start with a ready-made dataset. I recorded footage through our actual shelf camera, selected the useful frames, and created the dataset myself. I hand-labelled 57 frames across three classes: Base, Bottle, and Hand. I then fine-tuned YOLOv12n on those images and tested it against a held-out 40-frame clip.
Fifty-seven training frames is a small dataset, and I do not want to present it as anything else. It was enough to test the pipeline on our prototype, but a deployed system would need substantially more data across different products, shelves, lighting conditions, hands, and camera angles.
The weight sensor could tell us that something had moved, but not what caused it. The vision pipeline supplied that second check. It looked for a hand or bottle in the captured frames. If the detector found neither, a frame-difference fallback compared the first and last images for visible motion. A weight change that still could not be verified raised a discrepancy alert instead of being accepted silently.
We also used ByteTrack for aisle-side behaviour analytics. Track IDs existed only in memory and were discarded when the track closed. The prototype could record aggregate dwell and interaction events without keeping a persistent identity for the person in front of the shelf.
Changing the forecasting model
The first forecasting approach used Prophet. I evaluated it against a 30-day holdout, and it underperformed LightGBM by enough that continuing to tune it did not make sense. I moved the forecasting pipeline to LightGBM and used holdout mean absolute error to compare the models.
I trained a separate model bundle for each product using daily demand histories derived from the M5 Forecasting Accuracy dataset. Each bundle contained three quantile regressors: q10, q50, and q90. The dashboard could therefore show an uncertainty band around the median forecast instead of presenting one number as certain.
The models used thirteen lag, rolling-statistic, and calendar features. Those included the Friday-Saturday Saudi weekend and a Saudi-calendar demand multiplier. Training happened offline, while the local hub loaded the completed models for CPU inference. The q50 forecast also fed the pricing engine, where fixed rules limited how far a recommendation could move from the catalogue price.
Why it ran locally
AUJ was designed to work on the retailer’s local network without depending on a cloud connection. The hub ran FastAPI, PostgreSQL, MQTT, forecasting, vision verification, and live dashboard updates. Keeping the processing local also meant that camera data did not need to leave the store.
The team
- Abdulaziz Aldharrab, Team Lead, AI and machine learning
- Abdulrahman Alhussein, testing and security
- Ameen Khan, firmware and hardware
- Malik Alnajjar, backend and data
- Salman AlOtaibi, frontend and user experience
Project status
AUJ is an academic proof of concept, not a production retail system. The prototype closed the loop from a physical shelf event to a verified inventory update, a forecast, and a live dashboard. It also made the limits visible: the vision dataset needs to grow, hardware has to be recalibrated for a new shelf, and aisle-side person detection needs a separate detector before that part can move beyond the prototype.