How We Built Sentiment Analysis for Open-Text Responses

Open-text survey responses contain the richest feedback, but they don't scale — nobody wants to manually read ten thousand free-text answers. Automated sentiment analysis turns that unstructured text into a signal you can track and filter on.

Choosing an Approach

We evaluated three approaches: a simple keyword/lexicon-based classifier, a traditional machine learning model trained on labeled survey data, and a modern language model API. The lexicon approach was fast but too brittle for sarcasm and mixed sentiment. We landed on a hybrid: a fine-tuned lightweight model for the common case, with escalation to a more powerful model API for ambiguous or borderline responses.

Domain-Specific Training Data

Generic sentiment models trained on movie reviews or social media perform poorly on survey feedback, which uses different vocabulary and often expresses mixed sentiment ("the product is great but support was slow"). We fine-tuned on a labeled dataset of actual survey responses to better capture this domain-specific pattern.

Handling Mixed Sentiment

Many real responses aren't purely positive or negative — they praise one aspect and criticize another in the same sentence. Rather than forcing a single label, we return a sentiment score on a continuous scale alongside the discrete label, so a genuinely mixed response scores near the middle rather than being misclassified as neutral.

Processing at Scale

Sentiment scoring runs asynchronously as a background job immediately after a response is submitted, rather than blocking the submission itself. This keeps the respondent-facing experience fast while still surfacing sentiment scores in the dashboard within seconds for most responses.

Evaluating Model Accuracy Over Time

We maintain a held-out validation set of manually labeled responses that's never used for training, and periodically re-score it against the current production model to track accuracy drift. As survey topics and customer vocabulary evolve, model performance can silently degrade without an ongoing evaluation process to catch it — treating a sentiment model as a one-time build rather than an ongoing maintenance commitment is a common mistake.

Where It Falls Short

Sarcasm, idioms, and culturally specific expressions still trip up automated sentiment analysis more often than we'd like. We treat automated sentiment as a triage and prioritization tool — flagging clusters worth human attention — rather than a fully autonomous replacement for reading real feedback.