Assessing Problem-Solving Skills in Engineering Design-Based STEM Projects Using a Three-Tier Assessment Instrument
DOI:
https://doi.org/10.17509/wapfi.%22v%2211%22i%222.1464Keywords:
Engineering design-based projects, Problem-Solving Skills, STEM projects, Three-tier assessmentAbstract
Problem-solving skills are essential competencies in engineering design-based STEM learning, yet conventional assessments often fail to capture students' conceptual understanding, reasoning, and analytical explanation comprehensively. This study aimed to develop and implement a three-tier assessment instrument for assessing undergraduate students' problem-solving skills in interdisciplinary STEM projects. The study employed the ADDIE development model. The instrument was validated by three experts using Aiken's V coefficient and subsequently implemented with 75 undergraduate physics education students engaged in projects integrating Basic Technology Education and Basic Electronics courses. The three-tier assessment instrument comprised three response levels addressing conceptual understanding, logical reasoning, and analytical explanation. The content validation results indicated a very high level of validity. During implementation, the instrument successfully assessed problem-solving skills across these dimensions. Students who developed functional and high-quality engineering prototypes consistently demonstrated more accurate conceptual understanding, more coherent scientific reasoning, and deeper analytical explanations than those whose prototypes exhibited limited functionality. The three-tier assessment also provided diagnostic evidence of students' engineering thinking by revealing strengths and weaknesses unidentified by conventional assessments. These findings indicate that the proposed instrument is not only valid but also effective in providing comprehensive diagnostic information to support authentic assessment in engineering design-based STEM learning.
Downloads
References
Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45 (1), 131–142. https://doi.org/10.1177/0013164485451012
Aldila, C., et al. (2023). Development of educational products using the ADDIE model. International Journal of Educational Research Review, 5 (2), 356-368. https://doi.org/10.23887/ijerr.v5i2.50014
Anam, R. S., & Gumilar, S. (2024). A Systematic Literature Review of Tier Tests as Diagnostic Tools in Specific Areas of Science. Tadris: Jurnal Keguruan dan Ilmu Tarbiyah, 9 (1), 255–270. https://doi.org/10.24042/tadris.v9i1.17717
Auliya, H. R., Christianti, M., & Hidayat, B. (2025). Development of Multi-Tier Diagnostic Tests for Primary Schools: A Systematic Literature Review. Jurnal Prima Edukasia, 13 (1), 59–74. https://doi.org/10.21831/jpe.v13i1.76906
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. In Frontiers in Public Health 6 (149). https://doi.org/10.3389/fpubh.2018.00149
Docktor, J. L., Strand, N. E., Mestre, J. P., & Ross, B. H. (2015). Conceptual problem solving in high school physics. Physical Review Special Topics - Physics Education Research, 11 (2), 1–13. https://doi.org/10.1103/PhysRevSTPER.11.020106
Fang, S. C., Yang, K. L., & Fan, S. C. (2025). A conceptual framework for assessing transdisciplinary STEM practices. Research in Science & Technological Education, 43 (1), 272–293. https://doi.org/10.1080/02635143.2023.2264781
Gavrilas, L., & Kotsis, K. T. (2025). Integrating learning theories and innovative pedagogies in STEM education: A comprehensive review. Eurasian Journal of Science and Environmental Education, 5 (1), 11-17. https://doi.org/10.30935/ejsee/16538
Irmak, M., İnaltun, H., Ercan-Dursun, J., Yanis-Kelleci, H., & Yürük, N. (2023). Development and application of a three-tier diagnostic test to assess pre-service science teachers' understanding on work-power and energy concepts. International Journal of Science and Mathematics Education, 21(1), 159–185. http://dx.doi.org/10.1007/s10763-021-10242-6
Iwuanyanwu, P. N. (2020). Nature of problem-solving skills for 21st Century STEM Learners: What teachers need to know. Journal of STEM Teacher Education, 54 (1). https://doi.org/10.30707/JSTE55.1/MMDZ8325
Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. In International Journal of STEM Education, 3 (1). https://doi.org/10.1186/s40594-016-0046-z
Khamhaengpol, A., Sriprom, M., & Chuamchaitrakool, P. (2021). Development of STEAM activity on nanotechnology to determine basic science process skills and engineering design process for high school students. Thinking Skills and Creativity, 39, 100796. https://doi.org/10.1016/J.TSC.2021.100796
Kwon, H., & Lee, Y. (2025). A meta-analysis of STEM project-based learning on creativity and higher-order thinking. International Journal of STEM Education, 12 (5). https://doi.org/10.1186/s40594-025-00529-3
Maric, D., Fore, G. A., Nyarko, S. C., et al. (2023). Measurement in STEM education research: A systematic literature review of trends in the psychometric evidence of scales. International Journal of STEM Education, 10 (39), 1-22. https://doi.org/10.1186/s40594-023-00430-x
Maries, A., & Singh, C. (2023). Helping Students Become Proficient Problem Solvers Part I: A Brief Review. Education Sciences, 13 (2), 1–21. https://doi.org/10.3390/educsci13020156
Roslina, R., Samsudin, A., & Liliawati, W. (2022). Effectiveness of Project Based Learning Integrated STEM in Physics Education (STEM-PJBL): Systematic Literature Review (SLR). Phenomenon : Jurnal Pendidikan MIPA, 12 (1). https://doi.org/10.21580/phen.2022.12.1.11722
Susanti, E., Maulidah, R., & Makiyah, Y. S. (2021). Analysis of Problem-Solving Ability of Physics Education Students in STEM-Based Project-Based Learning. Journal of Physics: Conference Series. https://doi.org/10.1088/1742-6596/2104/1/012005
Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research instruments in science education. Research in Science Education, 48 (6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Thibaut, L., et al. (2018). Integrated STEM education: A systematic review of instructional practices. European Journal of STEM Education, 3 (1), 1-12. https://doi.org/10.20897/ejsteme/85525
Yang, K. L., Fang, S. C., & Fan, S. C. (2025). Development and validation of an instrument for assessing secondary students’ transdisciplinary STEM practices. International Journal of STEM Education 12 (5), 1-16. https://doi.org/10.1186/s40594-025-00529-3
Zakariya, Y. F. (2022). Cronbach's alpha in mathematics education research: Its appropriateness, overuse, and alternatives in estimating scale reliability. Frontiers in Psychology, 13:1074430. https://doi.org/10.3389/fpsyg.2022.1074430
Published
Data Availability Statement
The data that support the findings of this study are not publicly available due to confidentiality agreements but are available from the corresponding author upon reasonable request
Issue
Section
Categories
License
Copyright (c) 2026 Wahana Pendidikan Fisika
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The authors submitting a manuscript to this journal agree that, if accepted for publication, copyright publishing of the submission shall be assigned to The Journal Wahana Pendidikan Fisika (WAPFI). However, even though the journal asks for a copyright transfer, the authors retain (or are granted back) significant scholarly rights.
The Journal Wahana Pendidikan Fisika by https://ejournal.upi.edu/index.php/WapFi/ is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
