L4QAUCT HONOURSPrepare the task
Qualitative data, an analysis schema and taxonomies inform prompt formatting and workflow logic.
Contribution 02 / Sentiment Analysis
Look beyond the overall sentiment.
A comparison of single-pass and modular multi-agent workflows for document sentiment, aspect categories and aspect terms.
Research focus
How does separating sentiment tasks affect performance at different levels of analysis?
Context & contribution
One document can express different attitudes towards different aspects of an experience. This contribution compares a unified request with a sequence of specialised agents, examining both overall sentiment and the aspects or terms associated with it.
Both pipelines use quantized Qwen2.5-32B-Instruct with task examples and strict JSON contracts. A unified request is compared with three sequential specialists; structured outputs support review and traceable human corrections.
The approach
How it works.
Pipeline A: single pass
A single LLM request handles document sentiment (TSA), aspect categories (ACSA) and aspect terms (ATSA) together.
Pipeline B: specialised agents
Three sequential agents classify document polarity, map aspect categories and extract aspect terms.
Format and review
Outputs are formatted and checked against their contract, then made available for manual review and adjustments.
Two alternative pipelines share the same compute and inference layer, output processing and human-review stage.
Open full-size diagramEvaluation
What the results show.
The MAMS benchmark: 400 documents for sentiment and category analysis, with a 35-document subset for aspect-term evaluation.
Pipeline APipeline B
TSA · document sentiment
Macro-F1 · higher is betterACSA · aspect categories
Micro-F1 · higher is betterATSA · exact aspect term
Micro-F1 · higher is betterDiagnostic · Jaccard ≥ 0.5
Micro-F1 · overlap matchingView all values as a table
| Measure | Pipeline A | Pipeline B |
|---|---|---|
| TSA · document sentiment | 61.03% | 60.11% |
| ACSA · aspect categories | 44.80% | 45.77% |
| ATSA · exact aspect term | 41.12% | 45.32% |
| Diagnostic · Jaccard ≥ 0.5 | 49.53% | 52.22% |
Reading the resultsThe modular pipeline improved exact aspect-term micro-F1 by 4.20 percentage points and category micro-F1 by 0.97 points. Document sentiment macro-F1 declined by 0.92 points.
Source: Sithokomele Nxumalo’s final paper (PDF), Table 2 and Sections 3.1, 3.5 and 5.5.
Scope & limitations
What to keep in mind.
- Results cover one model and one benchmark. The 35-document term subset limits generalisation; no significance tests accompany the reported gains.
- Document-level reference sentiment was derived from aspect labels, rather than separately annotated. Pipeline A produced 399 valid TSA outputs, compared with 400 for B.
- The Jaccard ≥ 0.5 diagnostic allows partial term overlap. It helps explain phrase-boundary errors but is separate from exact-match performance.
More about the evaluation
TSA uses macro-F1; ACSA and ATSA use micro-F1. Overall sentiment reference labels use majority vote over aspect polarities, with ties mapped to neutral. Both pipelines used the same quantized model and temperature 0. Human review was demonstrated, but not assessed in a dedicated usability study.
Supporting material
All project resourcesRead the final paper.
Sentiment Analysis final paper
Sithokomele Nxumalo · Sentiment Analysis engine.