Metrics Metrics Metrics! ======================== What metrics are useful? What uses do we have? Justification ------------- Justify existence, justify to corporation, justify resources - consequences of not doing so very real - what is project health? how to improve? Drive improvements ------------------ Have to measure to improve - Contributor diversity - Community diversity (e.g. women in tech) Discussion ========== Metrics have different points at different lifecycle points (sales, pr, recruiting) "Define why community exists, then know what to measure" Metrics need to have some potential effect on decision-making; Avoid geeking out on numbers/graphs Can we access qualitative information too? Stories from users different but often quite useful. Regularly record first-person impressions over time. Surveys ------- Yearly surveys as source for reflection (e.g. State of Clojure) Surveys might be useful too with new contributors. One participant had bad experience with surveys. Better to include data collection with registration. What other communities have experience with metrics that we could learn from? To increase response rate: see if the use of the survey data has a value for the participant. Enhance aggregate quantitative stats with a selected subset of qualitative stats. Specific useful metrics ----------------------- Visitor count to a web site: measure effectiveness to know what works and what doesn't One way to think of metrics: "sales funnel" to measure attrition/engagement at different points in contributor lifecycles OpenStack -- percentage of commits from different companies (internal/external) User group -- besides "how did you hear of us", number of job referrals from community members, income before/after to justify group to participants Recruits as a result of open source contributions; also easier on-boarding (money saved) Not successful so far: page views, retweets, stars, other social media metrics. Not connectable to bottom line. Not successful: independent metrics. However historic on any metrics for individuals was really useful -- can be source for "exit interviews" too. Diversity ========= How to gather diversity metrics? How practically to gather gender/ethnicity/languages? In Ubuntu, Ubuntu women project tracked these details by hand; don't be afraid to use manual techniques for smaller groups Include metrics in event signups, include metric-taking as part of the process Be clear about use for metric (trust) Metrics can justify extra support e.g. to first-time speakers / participants at events Data overload ------------- Can you have too much data? Maybe :) Also selection bias. Being intentional about collection seems to be a good starting point. Important to trial the data collection also. Cohort analysis: track cohorts (e.g. new contributors who started January 2017) over time. Ethics ------ Is it ok to collect data for one purpose and use it for a different purpose? What considerations are here? Legal issues too in some entries. Essential issue to keep promise made to user and trust of user. Anonymization also a potential strategy. E.g. OpenStack raw user data not released to OpenStack open source community. Links ===== - Facebook's linking of github/fb accounts; did they open-source some tools? - Sarah Sharp's [FOSS Heartbeat](https://sarahsharp.github.io/foss-heartbeat/) - Carsten Wade mailing list - Slack+FB group: ["We are developer evangelist"](https://evangelistcollective.github.io/) - [SpinachCon](https://www.fsf.org/blogs/community/the-0th-spinachcon)
