gh-lyndonkl-claude/skills/cognitive-design/resources/evaluation-tools.md

Evaluation Tools

This resource provides systematic checklists and frameworks for evaluating designs against cognitive principles.

Tools covered:

1. Cognitive Design Checklist (general interface/visualization evaluation)
2. Visualization Audit Framework (4-criteria data visualization quality assessment)

---

Why Systematic Evaluation

WHY This Matters

Core insight: Cognitive design has multiple dimensions - visibility, hierarchy, chunking, consistency, feedback, memory support, integrity. Ad-hoc review often misses issues in one or more dimensions.

Benefits of systematic evaluation:

• Comprehensive coverage: Ensures all cognitive principles checked
• Objective assessment: Reduces subjective bias
• Catches issues early: Before launch or during design critiques
• Team alignment: Shared criteria for quality
• Measurable improvement: Track fixes over time

Mental model: Like a pre-flight checklist for pilots - systematically verify all critical systems before takeoff.

Without systematic evaluation: missed cognitive issues, inconsistent quality, user confusion that could have been prevented.

Use when:

• Conducting design reviews/critiques
• Evaluating existing designs for improvement
• Quality assurance before launch
• Diagnosing why design feels "off"
• Teaching/mentoring cognitive design

---

What You'll Learn

Two complementary tools:

Cognitive Design Checklist: General-purpose evaluation for any interface, visualization, or content

• Quick questions across 6 dimensions
• Suitable for any design context
• 10-15 minutes for thorough review

Visualization Audit Framework: Specialized 4-criteria assessment for data visualizations

• Clarity, Efficiency, Integrity, Aesthetics
• Systematic quality scoring
• 15-30 minutes depending on complexity

---

Why Cognitive Design Checklist

WHY This Matters

Purpose: Catch glaring cognitive problems before they reach users.

Coverage areas:

1. Visibility & Comprehension (can users see and understand?)
2. Visual Hierarchy (what gets noticed first?)
3. Chunking & Organization (fits working memory?)
4. Simplicity & Clarity (extraneous elements removed?)
5. Memory Support (state externalized?)
6. Feedback & Interaction (immediate responses?)
7. Consistency (patterns maintained?)
8. Scanning Patterns (layout leverages F/Z-pattern?)

Mental model: Like a doctor's diagnostic checklist - systematically check each vital sign.

---

WHAT to Check

1. Visibility & Immediate Comprehension

Goal: Core message/purpose graspable in ≤5 seconds

Checklist:

• Can users identify the purpose/main message within 5 seconds? (5-second test)
• Is important information visible without scrolling (above fold)?
• Is text/content legible? (sufficient size, contrast, line length)
• Are interactive elements distinguishable from static content?

Test: 5-second test (show design, ask what they remember). Pass: Identify purpose. Fail: Remember decoration or miss point. Common failures: Cluttered layout, poor contrast, content buried below fold. Fix priorities: CRITICAL (contrast), HIGH (5-second clarity), MEDIUM (hierarchy)

---

2. Visual Hierarchy

Goal: Users can distinguish primary vs secondary vs tertiary content

Checklist:

• Is visual hierarchy clear? (size, contrast, position differentiate importance)
• Do headings/labels form clear levels? (H1 > H2 > H3 > body)
• Does design pass "squint test"? (important elements still visible when blurred)
• Are calls-to-action visually prominent?

Test: Squint test (blur design). Pass: Important elements visible when blurred. Fail: Everything same weight. Common failures: Everything same size, primary CTA not distinguished, decoration more prominent than data. Fix priorities: HIGH (primary not prominent), MEDIUM (heading hierarchy), LOW (minor adjustments)

---

3. Chunking & Organization

Goal: Information grouped to fit working memory capacity (4±1 chunks, max 7)

Checklist:

• Are long lists broken into categories? (≤7 items per unbroken list)
• Are related items visually grouped? (proximity, backgrounds, whitespace)
• Is navigation organized into logical categories? (≤7 top-level items)
• Are form fields grouped by relationship? (personal info, account, preferences)

Test: Count ungrouped items. Pass: ≤7 items or clear grouping. Fail: >7 items ungrouped. Common failures: 15+ flat navigation, 30-field ungrouped form, 20 equal-weight metrics. Fix priorities: CRITICAL (>10 ungrouped), HIGH (7-10 ungrouped), MEDIUM (clearer groups)

---

4. Simplicity & Clarity

Goal: Every element serves user goal; extraneous elements removed

Checklist:

• Can you justify every visual element? (Does it convey information or improve usability?)
• Is data-ink ratio high? (maximize ink showing data, minimize decoration)
• Are decorative elements eliminated? (chartjunk, unnecessary lines, ornaments)
• Is terminology familiar or explained? (no unexplained jargon)

Test: Point to each element, ask "What purpose?" Pass: Every element justified. Fail: Decorative/unclear elements. Common failures: Chartjunk (3D, backgrounds, excess gridlines), jargon, redundant elements. Fix priorities: HIGH (decoration competing with data), MEDIUM (unexplained terms), LOW (minor simplification)

---

5. Memory Support

Goal: Users don't need to remember what could be shown (recognition over recall)

Checklist:

• Is current system state visible? (active filters, current page, progress through flow)
• Are navigation breadcrumbs provided? (where am I, how did I get here)
• For multi-step processes, is progress shown? (wizard step X of Y)
• Are options presented rather than requiring recall? (dropdowns vs typed commands)

Test: Identify what users must remember. Ask "Could this be shown?" Pass: State visible. Fail: Relying on memory. Common failures: No active filter indication, no progress indicator, hidden state. Fix priorities: CRITICAL (lost in flow), HIGH (critical state invisible), MEDIUM (minor memory aids)

---

6. Feedback & Interaction

Goal: Every action gets immediate, clear feedback

Checklist:

• Do all interactive elements provide immediate feedback? (hover states, click feedback)
• Are loading states shown? (spinners, progress bars for waits >1 second)
• Do form fields validate inline? (immediate feedback, not after submit)
• Are error messages contextual? (next to problem, not top of page)
• Are success confirmations shown? ("Saved", checkmarks)

Test: Click/interact. Pass: Feedback within 100ms. Fail: No feedback or delayed >1s without indicator. Common failures: No hover states, no loading indicator, errors not contextual, no success confirmation. Fix priorities: CRITICAL (no feedback for critical actions), HIGH (delayed without loading), MEDIUM (missing hover)

---

7. Consistency

Goal: Repeated patterns throughout (terminology, layout, interactions, visual style)

Checklist:

• Is terminology consistent? (same words for same concepts)
• Are UI patterns consistent? (buttons, links, inputs styled uniformly)
• Is color usage consistent? (red = error, green = success throughout)
• Are interaction patterns predictable? (click/tap behavior consistent)

Test: List similar elements, check consistency. Pass: Identical styling/behavior. Fail: Unjustified variations. Common failures: Inconsistent terminology ("Email" vs "E-mail"), visual inconsistency (button styles vary), semantic inconsistency (red means error and negative). Fix priorities: HIGH (terminology), MEDIUM (visual styling), LOW (minor patterns)

---

8. Scanning Patterns

Goal: Layout leverages predictable F-pattern or Z-pattern scanning

Checklist:

• Is primary content positioned top-left? (where scanning starts)
• For text-heavy content, does layout follow F-pattern? (top horizontal, then down left, short mid horizontal)
• For visual-heavy content, does layout follow Z-pattern? (top-left to top-right, diagonal to bottom-left, then bottom-right)
• Are terminal actions positioned bottom-right? (where scanning ends)

Test: Trace eye movement (F/Z pattern). Pass: Critical elements on path. Fail: Important content off path. Common failures: Primary CTA bottom-left (off Z-pattern), key info middle-right (off F-pattern), patterns ignored. Fix priorities: MEDIUM (CTA off path), LOW (secondary optimization)

---

Why Visualization Audit Framework

WHY This Matters

Purpose: Comprehensive quality assessment for data visualizations across four independent dimensions.

Key insight: Visualization quality requires success on ALL four criteria - high score on one doesn't compensate for failure on another.

Four Criteria:

1. Clarity: Immediately understandable and unambiguous
2. Efficiency: Minimal cognitive effort to extract information
3. Integrity: Truthful and free from misleading distortions
4. Aesthetics: Visually pleasing and appropriate

Mental model: Like evaluating a car - needs to be safe (integrity), functional (efficiency), easy to use (clarity), and pleasant (aesthetics). Missing any dimension makes it poor overall.

Use when:

• Evaluating data visualizations (charts, dashboards, infographics)
• Choosing between visualization alternatives
• Quality assurance before publication
• Diagnosing why visualization isn't working

---

WHAT to Audit

Criterion 1: Clarity

Question: Is visualization immediately understandable and unambiguous?

Checklist:

• Is main message obvious or clearly annotated?
• Are axes labeled with units?
• Is legend clear and necessary? (or use direct labels if possible)
• Is title descriptive? (conveys what's being shown)
• Are annotations used to guide interpretation?
• Is chart type appropriate for message?

5-Second Test:

• Show visualization for 5 seconds
• Ask: "What's the main point?"
  • Pass: Correctly identify main insight
  • Fail: Confused or remember decorative elements instead

Scoring:

• 5 (Excellent): Main message graspable in <5 seconds, perfectly labeled
• 4 (Good): Clear with minor improvements needed (e.g., better title)
• 3 (Adequate): Understandable but requires effort
• 2 (Needs work): Ambiguous or missing critical labels
• 1 (Poor): Incomprehensible

---

Criterion 2: Efficiency

Question: Can users extract information with minimal cognitive effort?

Checklist:

• Are encodings appropriate for task? (position/length for comparison, not angle/area)
• Is chart type matched to user task? (compare → bar, trend → line, distribution → histogram)
• Is comparison easy? (common baseline, aligned scales)
• Is cross-referencing minimized? (direct labels instead of legend lookups)
• Are cognitive shortcuts enabled? (sorting by value, highlighting key points)

Encoding Check:

• Identify user task (compare, see trend, find outliers)
• Check encoding against Cleveland & McGill hierarchy
  • Pass: Position/length used for precise comparisons
  • Fail: Angle/area/color used when position would work better

Scoring:

• 5 (Excellent): Optimal encoding, zero wasted cognitive effort
• 4 (Good): Appropriate with minor inefficiencies
• 3 (Adequate): Works but more effort than necessary
• 2 (Needs work): Poor encoding choice (pie when bar would be better)
• 1 (Poor): Wrong chart type for task

---

Criterion 3: Integrity

Question: Is visualization truthful and free from misleading distortions?

Checklist:

• Do axes start at zero (or clearly note truncation)?
• Are scale intervals uniform?
• Is data complete? (not cherry-picked dates hiding context)
• Are comparisons fair? (same scale for compared items)
• Is context provided? (baselines, historical comparison, benchmarks)
• Are limitations noted? (sample size, data source, margin of error)

Integrity Tests:

1. Axis test: Does y-axis start at zero for bar charts? If not, is truncation clearly noted?

   • Pass: Zero baseline or explicit truncation note
   • Fail: Truncated axis exaggerating differences without disclosure

2. Completeness test: Is full relevant time period shown? Or cherry-picked subset?

   • Pass: Complete data with context
   • Fail: Selective dates hiding broader trend

3. Fairness test: Are compared items on same scale?

   • Pass: Common scale enables fair comparison
   • Fail: Dual-axis manipulation creates false correlation

Scoring:

• 5 (Excellent): Completely honest, full context provided
• 4 (Good): Honest with minor context improvements possible
• 3 (Adequate): Not misleading but could provide more context
• 2 (Needs work): Distortions present (truncated axis, cherry-picked data)
• 1 (Poor): Actively misleading (severe distortions, no context)

CRITICAL: Scores below 3 on integrity are unacceptable - fix immediately

---

Criterion 4: Aesthetics

Question: Is visualization visually pleasing and appropriate for context?

Checklist:

• Is visual design professional and polished?
• Is color palette appropriate? (not garish, suits content tone)
• Is whitespace used effectively? (not cramped, not wasteful)
• Are typography choices appropriate? (readable, professional)
• Does style match context? (serious for finance, friendly for consumer)

Important: Aesthetics should NEVER undermine clarity or integrity

Scoring:

• 5 (Excellent): Beautiful and appropriate, enhances engagement
• 4 (Good): Pleasant and professional
• 3 (Adequate): Acceptable, not ugly but not polished
• 2 (Needs work): Amateurish or inappropriate style
• 1 (Poor): Ugly or completely inappropriate

Trade-off Note: If forced to choose, prioritize Clarity and Integrity over Aesthetics

---

Using the 4-Criteria Framework

Process:

Step 1: Evaluate each criterion independently

• Score Clarity (1-5)
• Score Efficiency (1-5)
• Score Integrity (1-5)
• Score Aesthetics (1-5)

Step 2: Calculate average

• Average score = (Clarity + Efficiency + Integrity + Aesthetics) / 4
• Pass threshold: ≥3.5 average
• Critical failures: Any individual score <3 requires attention

Step 3: Identify weakest dimension

• Which criterion has lowest score?
• This is your primary improvement target

Step 4: Prioritize fixes

1. CRITICAL: Integrity < 3 (fix immediately - misleading is unacceptable)
2. HIGH: Clarity or Efficiency < 3 (users can't understand or use it)
3. MEDIUM: Aesthetics < 3 (affects engagement)
4. LOW: Scores 3-4 (optimization, not critical)

Step 5: Verify fixes don't harm other dimensions

• Example: Improving aesthetics shouldn't reduce clarity
• Example: Improving efficiency shouldn't compromise integrity

---

Examples of Evaluation in Practice

Example 1: Dashboard Review Using Checklist

Context: Team dashboard with 20 metrics, users overwhelmed and missing alerts

Checklist Results:

• ❌ Visibility: Too cluttered, 20 equal-weight metrics
• ❌ Hierarchy: Everything same size, alerts not prominent
• ❌ Chunking: 15 ungrouped metrics (exceeds working memory)
• ❌ Simplicity: Chartjunk (gridlines, 3D, gradients)
• ❌ Memory: No active filter indication
• ✓ Feedback: Hover states, loading indicators present
• ⚠️ Consistency: Mostly consistent, minor button variations
• ❌ Scanning: Key KPI bottom-right (off F-pattern)

Fixes: (1) Reduce to 3-4 primary KPIs top-left, group others. (2) Remove chartjunk, establish hierarchy. (3) Show active filters as chips. (4) Standardize buttons.

Outcome: Users grasp status in 5 seconds, find alerts immediately

---

Example 2: Bar Chart Audit Using 4 Criteria

Context: Q4 sales bar chart for presentation

Audit Scores:

• Clarity (4/5): Clear title/labels, direct bar labels. Minor: Could annotate top performer.
• Efficiency (5/5): Optimal position/length encoding, sorted descending, common baseline.
• Integrity (2/5 - CRITICAL): ❌ Y-axis starts at 80 (exaggerates differences), ❌ No historical context.
• Aesthetics (4/5): Clean, professional. Minor: Could use brand colors.

Average: 3.75/5 (barely passes). Critical issue: Integrity <3 unacceptable.

Fixes: (1) Start y-axis at zero or add break symbol + "Axis truncated" note. (2) Add Q3 baseline for context. (3) Annotate: "West region led Q4 with 23% increase."

After fixes: Clarity 5/5, Efficiency 5/5, Integrity 5/5, Aesthetics 4/5 = 4.75/5 Excellent