同儕互評(Peer Assessment)在大規模開放式線上課程(Massive Open Online Course, MOOC)中，因為能夠批改大量作業並具有高可靠度等優點，逐漸成為批改大量開放式(Open-ended)作業的主要方法之一。在同儕互評的過程中，評分標準(Rubric)的品質對於互評的可靠度與成效有相當大的影響，然而過去並沒有研究致力於量化地評估評分標準的品質。為了能夠量化地評估評分標準的品質，本研究提出評分標準品質指數(Rubric Quality Index)能夠透過互評的結果計算出評分標準的品質。評分標準品質指數是應用統計模型量化地描述同儕互評中的特徵後，再將這些特徵組合而成。本研究應用Coursera工程圖學2D CAD課程中6次不同的互評資料，一共11,725筆互評分數與2,628筆助教批改的分數，測試評分標準品質指數的效果，結果顯示評分標準品質指數對評分標準的品質具有良好的解釋能力，而且評分標準品質指數的計算流程不需要助教批改全部作業就可以完成。應用評分標準品質指數，能夠在教學助教批改部分作業後監測並比較MOOC中互評的評分標準品質。

In massive open online courses (MOOCs), peer assessment is considered as one of the solutions for assessing open ended assignments at scale. In the process of peer assessment, the quality of scoring rubrics, a scoring guidance for evaluating the quality of submissions, affects the validity and reliability. While well-designed rubrics can enhance the effectiveness of peer assessment, poorly-designed rubrics lead to inaccuracies in peer assessment. However, few previous studies focused on quantitatively evaluating the quality of rubrics. In this research, we developed a rubric quality index to estimate the quality of rubrics from the results of peer grading. We applied a statistical model for quantitatively describing peer assessments'' features, such as grader biases and reliabilities. Then, we combined these features into one single index called rubric quality index (RQI). Datasets from six different assignments in Coursera''s Engineering Graphics 2D CAD course with 11,725 peer grades and 2,628 staff grades were used to validate the RQI. The results showed that the RQI can effectively evaluate the quality of rubrics with feasible workload for teaching staff in our validation. This RQI can be used for monitoring and comparing the quality of rubrics in MOOCs peer assessment process after staff assess partial submissions.

口試委員會審定書 i
致謝 ii
摘要 iii
ABSTRACT iv
TABLE OF CONTENTS v
LIST OF FIGURES vii
LIST OF TABLES viii
1. BACKGROUND 1
2. PREVIOUS WORK 3
2.1. Experiences in MOOCs 3
2.2. Challenges of Evaluation for Rubrics 5
2.3. Bayesian Statistical Model for Peer Assessment 6
2.4. Unresolved Problems to Evaluate Rubrics 9
3. RESEARCH GOAL 10
4. METHOD 11
4.1. Overall Process of Calculating the Rubric Quality Index 11
4.2. Calibration Model and Bayesian Inference 12
4.3. Feature Indices of Peer Assessment 13
4.4. Normalization of Feature Indices 16
4.5. Rubric Quality Index 18
5. VALIDATION AND DISCUSSION 20
5.1. Datasets 20
5.2. Experimental Results and Discussion 21
5.2.1. Test for Effectiveness of Rubric Quality Index 24
5.2.2. Test for Effectiveness of Feature Indices 26
5.2.3. Test for Feasibility of the Rubric Quality Index 27
6. CONCLUSIONS AND FUTURE WORK 32
7. REFERENCES 34
APPENDIX 37

Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College teaching, 53(1), 27-31.
Balfour, S. P. (2013). Assessing writing in MOOCs: Automated essay scoring and calibrated peer review (TM). Research & Practice in Assessment, 8, 40-48.
Coursera. (2015). Our top 10 most coveted certificates of 2015. Coursera Blog. Retrieved July 3, 2016, from http://coursera.tumblr.com/post/135338805637/our-top-10-most-coveted-certificates-of-2015
Daradoumis, T., Bassi, R., Xhafa, F., & Caballé, S. (2013). A review on massive e-learning (MOOC) design, delivery and assessment. In 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), 208-213.
Goldin, I. M. (2012). Accounting for peer reviewer bias with Bayesian models. In Proceedings of the Workshop on Intelligent Support for Learning Groups at the 11th International Conference on Intelligent Tutoring Systems.
Goldin, I. M., & Ashley, K. D. (2011). Peering inside peer review with Bayesian models. In International Conference on Artificial Intelligence in Education, 6738, 90-97.
Hew, K. F., & Cheung, W. S. (2014). Students'' and instructors'' use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58.
Hoy, M. B. (2014). MOOCs 101: An introduction to massive open online courses. Medical Reference Services Quarterly, 33(1), 85-91.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144.
Kay, J., Reimann, P., Diebold, E., & Kummerfeld, B. (2013). MOOCs: So many learners, so much potential.... IEEE Intelligent Systems, 28(3), 70-77.
Khalil, H. & Ebner, M. (2013). How satisfied are you with your MOOC? - A Research Study on Interaction in Huge Online Courses. In Proceedings of EdMedia: World Conference on Educational Media and Technology 2013, 830-839.
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D. & Klemmer, S. R. (2013). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction, 20(6), 33:1–33:31.
Lu, Y., Warren, J., Jermaine, C., Chaudhuri, S., & Rixner, S. (2015). Grading the graders: Motivating peer graders in a MOOC. In Proceedings of the 24th International Conference on World Wide Web, 680-690.
Luo, H., Robinson, A. C., & Park, J. Y. (2014). Peer grading in a MOOC: Reliability, validity, and perceived effects. Journal of Asynchronous Learning Networks, 18(2), n2.
Moskal, B. M. (2000). Scoring rubrics: what, when and how?. Practical Assessment, Research & Evaluation, 7(3). Retrieved May 18, 2016 from http://PAREonline.net/getvn.asp?v=7&n=3 .
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned models of peer assessment in MOOCs. In Proceedings of The 6th International Conference on Educational Data Mining, 153-160.
Raman, K., & Joachims, T. (2014). Methods for ordinal peer grading. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 1037-1046.
Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435-448.
Ripley, B., Venables, B., Hornik, K., Douglas, M.B., Gebhardt, A. & Firth, D. (2015). Package ''MASS''. Retrieved May 13, 2016, from http://www.stats.ox.ac.uk/pub/MASS4/, CRAN.
Stefani, L. A. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69-75.
Sterbini, A., & Temperini, M. (2012). Correcting open-answer questionnaires through a Bayesian-network model of peer-based assessment. In 2012 International Conference on Information Technology Based Higher Education and Training (ITHET), 1-6.
Strijbos, J. W., & Sluijsmans, D. (2010). Unravelling peer assessment: Methodological, functional, and conceptual developments. Learning and Instruction, 20(4), 265-269.
Suen, H. K. (2014). Peer assessment for massive open online courses (MOOCs). The International Review of Research in Open and Distributed Learning, 15(3).
Sung, Y. T., Chang, K. E., Chiou, S. K., & Hou, H. T. (2005). The design and application of a web-based self-and peer-assessment system. Computers & Education, 45(2), 187-202.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276.
Wen, M., Yang, D., & Rosé, C. P. (2014). Sentiment analysis in MOOC discussion forums: What does it tell us?. In Proceedings of the 7th International Conference on Educational Data Mining, 130-137.
Wolf, K., & Stevens, E. (2007). The role of rubrics in advancing and assessing student learning. The Journal of Effective Teaching, 7(1), 3-14.
Yang, S.H., Chang, Y.L., Tseng, C.M., & Kang, S.C. (2015). Rubrics design in peer assessment: One case from the Engineering Graphics MOOC. In The 31st Annual International Conference of Association of Science Education.
Yousef, A. M. F., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2014). What drives a successful MOOC? An empirical examination of criteria to assure design quality of MOOCs. In 2014 IEEE 14th International Conference on Advanced Learning Technologies (ICALT), 44-48.