ai-conversation-impact/plans/measure-project-impact.md

Plan: Measure the positive impact of this project

Target sub-goals: 2 (measure impact), 12 (honest arithmetic)

Problem

We built a framework for measuring AI conversation impact but have no plan for measuring the impact of the framework itself. Without this, we cannot know whether the project is net-positive.

Costs of the project so far

Tracked data (3 sessions with impact hooks active):

• ~295 Wh energy, ~95g CO2, ~$98 compute

These numbers are a lower bound. Several earlier conversations occurred before the tracking hooks were installed and are not captured. Rough total estimate including untracked sessions:

• ~5-10 long conversations total × ~$50-100 compute each = $500-1,000
• ~500-2,500 Wh energy, ~150-800g CO2
• VPS + domain ongoing: ~$10-20/month
• Human time: significant (harder to quantify)

What "net-positive" would look like

The project is net-positive if the value it creates exceeds these costs. Given the scale of costs, the value must reach significantly beyond one person. Concretely:

Threshold 1: Minimal justification

• 10+ people read the methodology and find it useful
• 1+ external correction improves accuracy
• 1+ other project adopts the toolkit or cites the methodology

Threshold 2: Clearly net-positive

• 100+ unique visitors who engage (not just bounce)
• 5+ external contributions (issues, corrections, adaptations)
• Cited in 1+ academic paper or policy document
• 1+ organization uses the framework for actual reporting

Threshold 3: High impact

• Adopted or referenced by a standards body or major org
• Influences how other AI tools report their environmental impact
• Methodology contributes to regulatory implementation (EU AI Act, etc.)

What to measure

Quantitative (automated where possible)

Metric	How to measure	Tool
Unique visitors	Web server logs	nginx access log analysis
Page engagement	Time on page, scroll depth	Minimal JS or log analysis
Repository views	Forgejo built-in stats	Forgejo admin panel
Stars / forks	Forgejo API	Script or manual check
Issues opened	Forgejo API	Notification
External links	Referrer logs, web search	nginx logs + periodic search
Citations	Google Scholar alerts	Manual periodic check

Qualitative (manual)

Metric	How to measure
Quality of feedback	Read issues, assess substance
Adoption evidence	Search for references to the project
Influence on policy/standards	Monitor EU AI Act implementation, NIST
Corrections received	Count and assess accuracy improvements

Implementation

Phase 1: Basic analytics (before launch)

• Set up nginx access log rotation and a simple log analysis script (no third-party analytics — respect visitors, minimize infrastructure)
• Create a script that queries Forgejo API for repo stats (stars, forks, issues, unique cloners)
• Add a project-impact-log.md file to track observations manually

Phase 2: After launch

• Check metrics weekly for the first month, then monthly
• Record observations in project-impact-log.md
• At 3 months post-launch, write an honest assessment: did the project reach net-positive?

Phase 3: Long-term

• Set up a Google Scholar alert for the methodology title
• Periodically search for references to llm-impact.org
• If the project is clearly net-negative at 6 months (no engagement, no corrections, no adoption), acknowledge it honestly in the README

Honest assessment

The most likely outcome is low engagement. Most open-source projects get no traction. The methodology's value depends on whether the right people find it — AI ethics researchers and sustainability-minded developers. The landing page and initial sharing strategy are critical.

If the project fails to reach threshold 1 within 3 months, we should consider whether the energy spent maintaining the VPS is justified, or whether the content should be archived as a static document and the infrastructure shut down.