Metrics Metrics Metrics!

What metrics are useful? What uses do we have?

Justification

Justify existence, justify to corporation, justify resources - consequences of not doing so very real - what is project health? how to improve?

Drive improvements

Have to measure to improve

Contributor diversity
Community diversity (e.g. women in tech)

Discussion

Metrics have different points at different lifecycle points (sales, pr, recruiting)

"Define why community exists, then know what to measure"

Metrics need to have some potential effect on decision-making; Avoid geeking out on numbers/graphs

Can we access qualitative information too? Stories from users different but often quite useful. Regularly record first-person impressions over time.

Surveys

Yearly surveys as source for reflection (e.g. State of Clojure)

Surveys might be useful too with new contributors.

One participant had bad experience with surveys. Better to include data collection with registration.

What other communities have experience with metrics that we could learn from?

To increase response rate: see if the use of the survey data has a value for the participant.

Enhance aggregate quantitative stats with a selected subset of qualitative stats.

Specific useful metrics

Visitor count to a web site: measure effectiveness to know what works and what doesn't

One way to think of metrics: "sales funnel" to measure attrition/engagement at different points in contributor lifecycles

OpenStack -- percentage of commits from different companies (internal/external)

User group -- besides "how did you hear of us", number of job referrals from community members, income before/after to justify group to participants

Recruits as a result of open source contributions; also easier on-boarding (money saved)

Not successful so far: page views, retweets, stars, other social media metrics. Not connectable to bottom line.

Not successful: independent metrics. However historic on any metrics for individuals was really useful -- can be source for "exit interviews" too.

Diversity

How to gather diversity metrics? How practically to gather gender/ethnicity/languages?

In Ubuntu, Ubuntu women project tracked these details by hand; don't be afraid to use manual techniques for smaller groups

Include metrics in event signups, include metric-taking as part of the process

Be clear about use for metric (trust)

Metrics can justify extra support e.g. to first-time speakers / participants at events

Data overload

Can you have too much data? Maybe :) Also selection bias. Being intentional about collection seems to be a good starting point. Important to trial the data collection also.

Cohort analysis: track cohorts (e.g. new contributors who started January 2017) over time.

Ethics

Is it ok to collect data for one purpose and use it for a different purpose? What considerations are here? Legal issues too in some entries. Essential issue to keep promise made to user and trust of user. Anonymization also a potential strategy. E.g. OpenStack raw user data not released to OpenStack open source community.

CLSxAU17: Community metrics clsxau17_community_metrics