Shuja Abidi

Shuja Abidi

Research Scholar in AI and Robotics

Shuja Abidi's research interests are in AI and robotics, in five fields:

Alongside this research, Shuja has more than 20 years of industrial experience in military simulation, and has built commercial AI products. See the simulators and the AI products

1

Human-Robot Interaction

Can a robot tell when a person is confused, surprised or not paying attention, and respond more safely?

Status: a research proposal with a working prototype. The detection methods have not yet been validated on HRI datasets.

The project, Bidirectional Expectancy Violation for Risk-Aware Human-Robot Collaboration, looks for violations in both directions:

  • Personal Deviation Violation (PDV): the person drifts from their own normal behaviour, or misses a response the robot expected.
  • Human Expectation Violation (HEV): the person reacts to an unexpected robot action.

Each violation would be attributed to the person or the robot, and would select a graded safety response that can only tighten the robot's behaviour.

Work so far

  • A real-time prototype that tracks facial action units, gaze, head pose and voice, learns a person's baseline and flags deviations.
  • A FLAME 3D face and avatar pipeline for making synthetic personal data from a 2-minute video.
  • A review of more than ten recent doctoral theses and the ERR@HRI challenge papers on robot errors and human reactions.

Methods

  • Per-person change detection: Mahalanobis distance and CUSUM
  • Time-series classification: MiniRocket + Hydra
  • 2D (OpenFace) compared with 3D (FLAME) facial features

Planned evaluation datasets

REFLEX, ERR@HRI, BAD, HRI-confusion and UE-HRI

A table with four rows linking the onset and shape of a signal to its attribution and a response. A sharp reaction within 2 to 5 seconds of a robot action is HEV, so the robot retracts, stops or asks. A sustained reaction is HEV needing explanation. An absent expected response is PDV unawareness, so the robot holds its approach and cues attention. Slow change with no preceding robot action is PDV drift, so the robot adapts its pace and checks in.
The proposed logic: when a signal begins and how it changes suggest its cause and the robot's response. The signal shapes are illustrations.

2

JEPA

Learning by predicting hidden parts of an image in feature space, with no labels and no pixel reconstruction.

Status: a hands-on study, built one lesson at a time. I-JEPA and the Joint-Embedding Predictive Architecture idea are the work of their original authors at Meta and elsewhere.

This study works through the ideas behind JEPA by building them: why such models can collapse, predicting pixels compared with predicting features, a Vision Transformer from scratch, and then a small I-JEPA.

It then joined fields 2 and 3 in a side experiment, Jepa-CBCL-PR. It replaces CBCL-PR's ResNet-34 with frozen I-JEPA encoders, to ask how much the encoder matters for continual learning.

The strong encoder clearly helped. The small encoder trained here fell well behind on new classes, but still beat an untrained one, so it learned something that transfers.

Jepa-CBCL-PR on CIFAR-100

Frozen encoderAccuracy
Meta's I-JEPA74.23%
ResNet-3462.59%
Small I-JEPA, trained here26.67%
Untrained encoder10.53%

CBCL-PR, using all training images. An exploratory side experiment. The large encoders were trained on bigger images and far more data.

I-JEPA architecture. An image is split into context patches and hidden target blocks. A context encoder and a predictor try to match features from a target encoder, which is updated by a moving average. After training, the target encoder is kept to produce image features.
I-JEPA as built in the study, with CIFAR-10 settings.

3

CBCL

Cognitively-inspired continual learning: remembering classes as compact statistics, not stored images.

Status: a reproduction, done to understand the field. CBCL and CBCL-PR are the work of Dr Ali Ayub and co-authors. The results are preliminary, and the reproduction isn't finished.

Robots working with people need to keep learning, often from very few examples, without forgetting. Both methods were implemented from the thesis and papers, compared with the official CBCL code, and tested on CIFAR-100, Caltech-101 and CUBS-200-2011 against the FLB baseline. The code passes 17 hand-checked tests.

One lesson from this work: the voting formulas in the thesis and in the official code differ. With 5 images per class on Caltech-101, they gave 78.92% and 84.65% in these runs.

Next: finish the full benchmark, and run ConceptGraphs to learn how robots build maps of objects. After that, the aim is to add teaching by a person, and then a memory that can ask about discrepancies.

Reproduction on CIFAR-100

MethodThis workThesis
CBCL-PR56.76 ± 0.57%56.98%
CBCL53.16 ± 1.94%54.7%

5 images per class, 10 classes per step. CBCL-PR over 3 runs, and CBCL over 10.

CBCL and CBCL-PR architecture. Images of new classes pass through a frozen ResNet-34. Clustering stores centroids in long-term memory. CBCL predicts by a weighted vote of nearest centroids. CBCL-PR samples fake features from stored statistics to train a linear classifier.
CBCL and CBCL-PR, redrawn from Ayub's thesis to explain how they work.

4

Semantic Safety

Using what objects are to keep a robot safe: a cup of water must not pass over a laptop, but a dry sponge can.

Status: a partial reproduction in simulation, built one lesson at a time to understand the field. Semantically safe manipulation is the work of Brunke et al., RA-L 2025. There is no real robot, camera or LLM here yet, and the rules are typed by hand.

A filter that only avoids collisions can still break common-sense rules. This study rebuilds the idea in six lessons: a safety filter on a point robot, several constraints with superquadric shapes, semantic regions, the filter on a 7-joint Franka FR3 arm, whole-arm collisions and rotation, and finally a simulated kitchen.

One lesson from this work: zero violations came with a real cost. Many semantic runs stopped before the goal. The scripted motions can't steer around anything, and the protective spheres may be too large, but these causes haven't been separated yet.

The filter also runs in a web browser, with nothing to install, and matches the Python results on 41 of 41 checks.

Next: rules from an LLM, then perception from RGB-D, then testing moving objects and position errors.

Simulated kitchen, 40 runs per filter

FilterWorst violationsReached goal
No filter61.9%100%
Geometric42.0%95%
Semantic0.0%72.5%

Violations: % of steps breaking a rule, in the worst of 8 scene and object cases. Reached goal: the average across those cases. The rules, motions and layout are this study's own, so compare the pattern with the paper, not the values.

Robot paths towards a goal past a laptop, in side view and from above. With no filter and with sponge rules, every run passes over the laptop. With cup rules, no run passes over it: seen from above, the paths go around a region drawn around the laptop.
Same motions, different held object. The side view can look as if the cup's paths cross the region, but the view from above shows them going around it.

5

Robotics

Legged robots that see people, plan around them and walk there.

Status: all in simulation (MuJoCo), with no real robot yet. The humanoids walk with Unitree's pretrained policies. The research effort goes into perception and planning: deciding where to go.

A Unitree Go2 quadruped and three Unitree humanoids (G1, H1 and H1-2) each build a map from their own RGB-D camera and 360° lidar, track walking people, plan a path that respects personal space, and walk it. Every robot takes the same velocity command, so one planner drives them all.

For the Go2, three walking controllers are compared: a hand-written trot, an MIT Cheetah-style convex MPC rebuilt from Di Carlo et al. (2018) with my own QP solver, and a learned RL policy.

One lesson from this work: human-aware planning was worse than normal planning until a 360° lidar was added, because a forward camera misses people coming from the side.

Results in simulation

TestResult
People crossing the path, G1, 12 runshuman-aware: closest 0.96 m median, 0.49 m worst; normal: 0.61 m, one collision
Go2 ramps 5°–25°MPC crosses all; simple trot stuck from 15°
Demo paths, all four robots0 falls
The robot simulator window: a sidebar of buttons on the left, and a Unitree G1 humanoid walking among posts, boxes and three simulated people.
The simulator, with all controls in the sidebar: robot, mode, goals, objects and options such as walking people and human-aware planning.

Industrial experience: military simulation

More than 20 years of designing and building military training simulators, from hardware to software.

Shuja began developing military and medical training simulators in 2002 at Data Communication and Control, including tank, small-arms and car simulators, and built distributed simulation infrastructure and 3D graphics engines.

Since 2007, Shuja has been Chief Technology Officer of Soft Innovative Systems, leading platform selection, system architecture and technical design, and building the engineering team. The company's simulators pair realistic hardware with an instructor station that records, replays and scores each trainee.

  • 350+ simulators delivered
  • 11 product lines
  • 2 offices in Pakistan

Company figures, as stated on sispvt.com.

www.sispvt.com The home page of the Soft Innovative Systems website: Training that makes readiness measurable. Visit the company website

Projects

Each picture opens the project's page on the company website.

AI Products

Commercial AI in two areas: computer vision for fashion, and language models for business.

Virtual try-on, Vocal Eye (2021 – 2023)

AI that shows a person wearing a chosen garment by changing the clothes in their photo. The team built and trained its own implementation of CP-VTON (Characteristic-Preserving Virtual Try-On Network) and its successor, CP-VTON+.

  • Reading the photo: human parsing (DensePose), pose detection (OpenPose) and cloth segmentation (U-Net).
  • Fitting the garment: a geometric matching stage warps the clothing to the person's body with a thin-plate spline.
  • Making the image: a try-on stage blends the warped garment with the person through a composition mask, designed to keep the garment's texture and logos.
  • Also in the pipeline: masked-image composition and diffusion-based garment synthesis.

Methods: CP-VTON (Wang et al., ECCV 2018); CP-VTON+ (Minar et al., CVPR Workshops 2020).

AI assistants for business, Splutter (2024 – present)

A SaaS platform where companies build AI chatbots and voicebots and deploy them on their own websites within minutes. More than 100 companies use it. Shuja is the platform's architect and led its core AI integration.

  • Knowledge: companies load a knowledge base from websites, PDFs, text and manual entry.
  • Answers: retrieval-augmented generation (RAG) bases each reply on the company's own content, using OpenAI and other language models.
  • Models: AI models trained for the platform's assistants, and a bot configuration that lets models be swapped.
  • Reach: web, SMS, WhatsApp, Messenger and Slack, for customer support, sales and lead capture.

Experience

More than 25 years of building real-time systems that people use and train on.

Chief Technology Officer, Soft Innovative Systems

Leads platform selection, system architecture and technical design for simulation and software products. Built and leads the engineering team.

Technical Head, Splutter

Architect of an AI chatbot and voicebot SaaS platform used by more than 100 companies. Led the core AI integration, including RAG and OpenAI language models. More

Co-founder, Vocal Eye

Built virtual try-on with an in-house, trained implementation of CP-VTON and CP-VTON+, along with other AI products. More

Technical Head, Devaj Technology

Led delivery of a restaurant management system used by more than 2,000 clients in Pakistan, and a cloud retail management system.

System Architect and Team Lead, Data Communication and Control

Developed military and medical training simulators, distributed simulation using HLA and NDDS, and 3D graphics engines.

Research Assistant, Muhammad Ali Jinnah University

Lecturer, Hamdard University, Karachi

Skills

Human sensing
FLAME 3D face fitting, FlashAvatar, facial action units, gaze and head pose, openSMILE voice features, Whisper
AI and machine learning
OpenFace, MediaPipe, DensePose, OpenPose, U-Net, CP-VTON and CP-VTON+ virtual try-on, diffusion models, LLM integration (OpenAI), RAG, MiniRocket, Mahalanobis distance, CUSUM
Semantic safety and robot control
Control barrier function safety filters, quadratic programming (OSQP), superquadrics, MuJoCo, Franka FR3 inverse kinematics
Legged robots
MuJoCo, Unitree G1, H1, H1-2 and Go2, convex MPC, RL policies (ONNX), RGB-D and lidar mapping, A* and human-aware planning, ROS 2
Simulation and graphics
Unreal Engine, Unity, OpenGL, XNA/DirectX, HLA and NDDS, simulator hardware integration
Programming
Python, C#, C++, SQL, Next.js, Flutter, React Native

Contact

shujaabidi2020@gmail.com
Based in Pakistan. Full CV and references are available on request.