Student Process Data in Educational Assessment Wade Buckland,

Student Process Data in Educational Assessment Wade Buckland,

Student Process Data in Educational Assessment Wade Buckland, Research Scientist, HumRRO Cheryl Alcaya, Test Development Supervisor, Minnesota DOE Jennifer Dugan, Director of Statewide Testing, Minnesota DOE Monica Gribben, Senior Staff Research Scientist, HumRRO Introduction to Process Data June 28, 2017 Presented at: CCSSO Austin, TX Presenter: Wade Buckland

paper assessment digital assessment scoring/reporting speed adaptive capability timing universal design innovative items paper assessment digital

assessment scoring/reporting speed adaptive capability timing universal design innovative items process data process data user interactions/actions that can be observed/recorded

clickstreams (next, back, etc.) keyboard/touchscreen presses screen coordinates (mouse click) time stamping 5 process data user interactions/actions that can be observed/recorded clickstreams (next, back, etc.) keyboard/touchscreen presses

screen coordinates (mouse click) time stamping enter item exit item change volume change language click choice keyboard cut right click cut text-to-speech zoom play pause reach end of tutorial backspace

delete clear answer mouse cut mouse copy mouse paste menu Italic right click italic mouse underscore keyboard underscore menu underscore right click underscore mouse spell check menu spell check mouse misspellings menu thesaurus

right click thesaurus thesaurus replacement Item scroll response scroll stimulus scroll hide timer show timer 6 process data unit of analysis depends on inference time to an event

specific events an event in a given context workload to completion a sequence of events specific approaches to task completion enter item exit item change volume change language click choice keyboard cut right click cut text-to-speech zoom play

pause reach end of tutorial backspace delete clear answer mouse cut mouse copy mouse paste menu Italic right click italic mouse underscore keyboard underscore menu underscore right click underscore mouse spell check menu spell check

mouse misspellings menu thesaurus right click thesaurus thesaurus replacement Item scroll response scroll stimulus scroll hide timer show timer 7 process data handling of data is burdensome

record/generate validate transform filter enter item exit item change volume change language click choice keyboard cut right click cut text-to-speech zoom

qualify (i.e., score) analyze play pause reach end of tutorial backspace delete clear answer mouse cut mouse copy mouse paste menu Italic right click italic mouse underscore keyboard underscore

menu underscore right click underscore mouse spell check menu spell check mouse misspellings menu thesaurus right click thesaurus thesaurus replacement Item scroll response scroll stimulus scroll hide timer show timer 8 process data & timing

Estimate engagement, motivation, fluency, reading speed, possibility of cheating Inform item development, test assembly, and calibration sample 9 process data & universal design / usability

test prototypes of interface or modifications over time supplement usability tests/cognitive labs with quantitative data (e.g., time on task) examine universal design tools and accommodation usage of special populations 10 process data & universal design / usability

Microsoft Research Lumiere Project, 1993-1995 11 process data & universal design / usability Microsoft Research Lumiere Project, 1993-1995 12 process data & writing Research on how writing processes like dysfluency, macroediting, local editing, and planning related to scoring rubric levels (Zhang & Deane, 2015) Capture action and context information

Leijten & Van Waes, 2013 13 process data & innovative items student background data performance data process data 14

New Directions for Minnesota Science Assessments Cheryl Alcaya | Test Development Supervisor June 28, 2017 Today Current assessments: Assessments in grades 5, 8 and high school. Based on academic standards adopted in 2009. Summative, end-of year. Comprehensive, (assesses grade bands for 3-5 and 6-8); high school assessment measures life science standards. Online, 50% TEI including simulations, 50% selected response. 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us

16 Today How long have we been thinking and talking about an integrated assessment system? It feels like forever! Why arent we there yet? Policy issues Technical quality expectations Local control Etc. 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us

17 Possible Design Common Interim, classroom on-demandassessments assessments Shortened summative assessment 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 18

Tomorrow Minnesotas academic standards for Science will be revised in 2018-2019. Next Generation Science Standards? NGSS-like standards? 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 19 Accountability Assessment in a Multi-Dimensional World Can rich simulations and click-stream technology help us measure multidimensional learning (and still pass peer review)? Highlight Science and Engineering Practicesconduct experiments without lab equipment.

Highlight inquiryprovide data on students inferencing skills. Explore scientific phenomena otherwise inaccessible to students. 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 20 Pilot test of rich simulation tool School year 2017-2018. Instruments are primarily teaching and learning tools. Teachers can administer the tests on-demandat the conclusion of a related unit of instruction. Teachers have access to student profiles, with a goal to allow for more targeted intervention and enrichment.

02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 21 Pilot test of rich simulation tool School year 2017-2018. Instruments are primarily teaching and learning tools. Teachers can administer the tests on-demandat the conclusion of a related unit of instruction. Teachers have access to student profiles, with a goal to allow for more targeted intervention and enrichment. 02/21/2020

Leading for educational excellence and equity, every day for every one. | education.state.mn.us 22 The Policy Perspective of Process Data Jennifer Dugan| Director, Statewide Testing Policy Considerations Perception of collecting too much data? Subjects and grade levels? All? Only those best suited? Impact to program Educator buy-in

Fiscal? Timing is everything 02/21/2020 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 24 Can Process Data Lead to a Deeper Understanding of Student Performance? Presenter : Monica Gribben, HumRRO June 28, 2017

Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314-1578 | Phone: 703.549.3611 | www.humrro.org NAEP Digitally Based Assessments 2009 Interactive science tasks 2011 Writing 2014 Technology and Engineering Literacy 2017 Math, Reading, and Writing From: https://nces.ed.gov/nationsreportcard/dba/ 26

Overview Why did we explore the use of process data for NAEP? What did we do? How can you use process data? What do you need to know to collect and use process data? 27 Why is the NAEP program interested in process data?

Leverage the use of technology Inform item development Improve the student experience Improve understanding of student performance 28 Methodology Literature Review

Student assessment Education software Gaming and simulation Response time Eye tracking Data logging Interviews

Expert Review 29 Findings Using process data in large-scale assessments PISA exploratory research and secondary reporting Educational software, gaming, and research Using process data to inform NAEP item development and assessment design Student interactions versus expectations Accommodations use versus expectations Universal design features and tutorial or system design

Using process data to inform scoring and reporting Secondary reports 30 Findings (continued) Using task or process models Answer specific research-based questions Education software/games - specific environments and focused questions

Differences by NAEP subject Obvious elements (calculator, equation editor, passage navigation) Differences by NAEP item type No patterns 31 Recommendations for Using Process Data

Collect and store all process data elements Use cognitive expectations to develop models Identify meaningful sets of data elements Look for patterns Provide evidence of validity Understand use of accommodations better Develop task models Improve the usability of the assessment platform 32 Advancing the use of process data

Recognize that using process data requires resources Develop a common language Share resources 33 Questions

Recently Viewed Presentations

  • Figure 1.1 A telephone system.

    Figure 1.1 A telephone system.

    Parasitic resistor and inductor are ignored at low frequencies Circuit Symbol Practical Inductor Inductor Any conductor of electric current has inductive properties and may be regarded as an inductor In order to enhance the inductive effect, a practical inductor is...
  • MoneyCounts: A Financial Literacy Series

    MoneyCounts: A Financial Literacy Series

    A credit card is a powerful financial tool, especially for building your credit history. Though, it should be managed very carefully because it may cause you to spend more than what is allowed in your budget. It is necessary to...
  • Game evaluation by Evan Cole Basic Information Company:

    Game evaluation by Evan Cole Basic Information Company:

    Sound and Music. The game is comprised of orchestra pieces. Banjo and Kazooie make different noises for every move they make. Each creature have their own noises, they are louder as you are closer to them. Jinjos make a whistling...
  • JLEIC Design Thomas Jefferson National Accelerator Facility, Newport

    JLEIC Design Thomas Jefferson National Accelerator Facility, Newport

    JLEIC Design by JLEIC Collaboration. Abstract . The recent NAS Review concluded an electron-ion collider are significant to advancing our understanding of the atomic nuclei that make up all visible matter in the universe.
  • Literary Elements

    Literary Elements

    (Shrek) EX: Things are not always as they seem. **In analysis, theme is given as a complete sentence= Love inspires sacrifice. Plot. DEF: how the action of the story is presented. Exposition, rising action, climax, falling action, resolution. plot triangle.
  • A corpus of conference presentations at the University of ...

    A corpus of conference presentations at the University of ...

    > i'm sorry but teachers get anxious too, and they are very proficient, most of the time. < READING >anxiety has negati- negative impact on language performance< /READING >. we found as a matter of fact that sometimes the anxiety...
  • Brush strokes in Writing: Painting with Absolutes

    Brush strokes in Writing: Painting with Absolutes

    Brush strokes in Writing: Painting with Absolutes. What Skills are we practicing? using a variety of correctly structured sentences (e.g., compound, complex, compound-complex). Use and understand complex grammatical structures in writing (an absolute phrase is a complex structure)
  • FAucS: An FCC Spectrum Auction Simulator for Autonomous ...

    FAucS: An FCC Spectrum Auction Simulator for Autonomous ...

    Language Games and Other Meaningful Pursuits Michael L. Littman [email protected] AT&T Labs Research (sort of)