## 1. Introduction

There has been a widespread international movement toward integrated STEM (Science, Technology, Engineering, and Mathematics) education into school science curricula. In many international science education reform document, engineering integration is considered as a useful and potential pedagogical approach to preparing future STEM workers (NRC, 2009;NRC, 2012;NGSS Lead States, 2013). Science educators and researchers agree that the key of STEM education is integrating “engineering design” into science instruction (Roehrig et al., 2012;Roehrig, 2017).

Researches show that integrating engineering into K-12 science classrooms improve students’ science literacy, positive self-efficacy, career aspirations in science and engineering as well as real-world problem solving skills (e.g. Apedoe, et al., 2008;Brophy, et al., 2008;Burgin, McConnell, & Flowers III, 2014). While there has been increasing evidence about the positive effects of engineering integrated science instruction, engineering integration in school science is still in its infancy in South Korea science classroom.

Government of South Korea also recognizes the importance of STEM education for future citizens (Korea Institute of S&T Evaluation and Planning, 2014). Recently released National reformed science curriculum addressed the needs of integrating engineering in school science instruction (Ministry of Education, 2015). However, there is no clear direction or guidance for science teachers how to integrated engineering in science lesson. There is no consistent and clear definition about engineering integration and the scope of engineering integration in science lesson is not specific enough (Lee & Nam, 2018). Sim et al. (2015) argue that the STEM instruction criteria given by Korea Foundation for the Advancement of Science and Creativity (KOFAC) are not easy to understand because it includes too many sub criteria that is not clearly related to each other (Lee & Nam, 2018).

Researchers and national documents seem to agree that the key of engineering integration is engineering design (Moon, 2008;Moore et al., 2015;NGSS Lead States, 2013). ‘Engineering Design’ is considered as a set of skill or strategy in solving engineering problem context. Three common components of engineering design skill include: 1) defining the problem, 2) design solution, and 3) optimizing the solution (NGSS Lead States, 2013). Engineering design is often considered as a sequential problem-solving process. However, the components of engineering design “do not always follow in order” and a problem-solver can “redefine the problem or generate new solutions to replace an idea that just isn’t working out” (NGSS Lead States, 2013, Appendix I, p. 2). In other words, at any point of engineering design, a problem-solver can go back to any step of the engineering design process as needed. The flexible and iterative process of design solution allows learners to improve problem-solving skills by challenging them to generate a better solution (Kang & Nam, 2016;Roehrig et al., 2012).

In engineering design context, engineers face multiple failures. This process of multiple trial and errors is necessary to improve the quality of engineering product. Likewise, purposefully planned failure experiences are necessary for improving students’ problem solving skills and opportunity of applying science knowledge (Karpur, 2008). Researchers argue that this opportunity of resign solutions and science knowledge application for it were essential components of engineering based science instruction(Guzey et al., 2014;Moore et al., 2014;Nam et al., 2016; Roehrig, 2018). However, not many of the pre-college engineering integrated STEM lesson include clear criteria to judge students’ engineering product and application of science knowledge in students’ product (Guzey et al., 2014).

Lee and Nam (2018) analyzed 76 STEM science lesson units developed by Korea Foundation for the Advancement of Science & Creativity (KOFAC)’s support and found that the lesson units do not include complete engineering design process with science knowledge application. Only 5% of KOFAC’s STEM units include high quality engineering design integrated science instruction that include evaluation and redesign process (Lee and Nam, 2018). Consequently, there is little study about what students learn from the complete engineering design process with science knowledge application.

Thus, the primary purpose of the study is to understand what students learn from complete engineering design context with science knowledge application compared to incomplete engineering design without science knowledge application in South Korea science classroom. For this study, two different types of lesson units were developed (called Type I lesson units and Type II lesson units). In Type I lesson units, students learned-learn science knowledge and need to design solutions multiple times by applying science knowledge they have learned to meet the criteria for success. In Type II lesson units students learn science knowledge but the knowledge is not directly required to their design solutions. By comparing the effect of two different types of engineering design lesson units on students’ creative engineering problem solving propensity (CEPSP) (Kang and Nam, 2016) and their perceptions, this study shows how redesign experience with science knowledge application is critical for improving students’ problem solving skills in engineering context.

## 2. Method

This study presents mixed research methods using both qualitative and quantitative methods.

### 2.1. Research Background and Lesson Development

The study was conducted in a middle school in B metropolitan city, South Korea. The research subjects were divided into two groups and each group took one of the two types of engineering integrated science units (71 students participated in Type I lesson and 75 students participated in Type II lesson). Type I lesson include two lesson units with different topics, ‘Buoyancy’ and ‘Guitar’. In the Buoyancy lesson unit, students learned about the science concept of buoyancy first and were asked to develop a boat that could hold a certain weight of objects by applying the science concept they have learned. To develop a successful prototype, students needed to apply the concept of buoyancy in their design solution. In the Guitar lesson unit, students, learned about the science concept of wave and the relationship between sound wave and materials. To develop a simple guitar that could play a simple song with five codes, students compared sounds from different materials and checked the code of the sounds. In doing so, students needed to apply their knowledge of wave and sound to develop a successful prototype. Both ‘Buoyancy’ and ‘Guitar’ lesson units have 8 lesson hours. Type II lesson also has two different lesson units, ‘Environment’ and ‘Insulation’. In the ‘Environment’ lesson, students were asked to develop an environmentally friendly water bottle with eatable chemicals. In ‘Insulation’ lesson, students were asked to build a heat-insulating model house.

Each lesson unit was carefully designed for students learn scientific concepts and principles, and to design solutions in given limitations and constrains (budget and time). However, in Type I lesson units, students need to apply science knowledge they have learned in the unit to design solutions multiple times to meet the criteria for success. In Type II lesson units students learn science knowledge related to engineering design but the knowledge is not directly required to design and redesign solutions.

Both set of lesson units were checked its content validity based on quality STEM lesson criteria using 5 Likert scale (16 items in 6 categories; Purpose, Concept of STEM, Context, Creative design, Emotional Experience, Assessment) by 7 STEM education expert and experienced classroom teachers. The average rating of the experts for four lesson units were; ‘'Buoyancy (4.47)', 'Guitar (4.71)’, ‘Environment (4.60)’, and ‘Insulation (4.84)’.

### 2.2. Subject

The subjects of this study are 146 7th-grade middle school students (86 boys and 60 girls) during 2018 school year. According to their achievement level, 20% of boys are high, 60% are middle, and 20% are low achieving students and 20% of girls are high, 50% are middle, and 30% are low achieving students. In terms of achievement level, there is not much difference between boys and girls. All of the students worked in single-gender small groups. The students were grouped as Cohort I (N=71) and Cohort II (N=75). Cohort I students participated in Type I lessons and Cohort II students participated in Type II lessons. Each Cohort I and II included two classrooms (A and B). Two teachers taught one of two classrooms (classroom A or B) in Type I as well as Type II lessons.

### 2.3. Data collection

#### 2.3.1. Creative Engineering Problem Solving Propensity (CEPSP)

This study used an engineering problem solving skill assessment tool called, Creative Engineering Problem Solving Propensity (CEPSP) developed by Kang & Nam (2017). CEPSP is composed with 28 items in five categories (motivation, engineering design, engineering habits of mind, understandings of engineering and engineers, communication and collaboration skill). The exploratory factor analysis result showed that the reliability of each construct category was between .733 to .892., meaning that the instrument is reliable in terms of the higher structural validity (each item is categorized in an appropriate construct category) and the reliability of the total items was .906. The CEPSP instrument items were used to measure the impact of middle school students’ engineering design-based science unit on their CEPSP. The pre-post test results were analyzed by a paired t-test with the significance level of p< .05.

#### 2.3.2. Survey

Before and after each lesson unit, students were asked to answer open-ended survey questionnaire about their perceptions of engineering. The questionnaire includes items about perceptions of how much they apply prior knowledge and experience and complexity level of the given engineering problem, and perception of what kind of ability engineers need.

### 2.4. Data analysis

Students’ pre-post CEPSP data were statistically analyzed by paired T-test, using the statistical program, SPSS Statistics 21. A paired T-test was used to determine the difference between pre-post CEPSE in each lesson group. Student answers in open-ended items in survey questionnaire were analyzed qualitatively. The answers were carefully read by 2 researchers to develop initial theme. Based on the theme, students’ answers were categorized by each theme. When we analyze student responses, we found that a student’s response could include multiple responses that could be categorized in multiple anlaysis categories. So we count each student’ response in each applied categorizes. In addition, descriptive statistical analysis was used to present frequencies of the answers in the developed category by each theme. Table 1

## 3. Results

### 3.1. Comparison of students’ Creative Engineering Problem Solving Propensity (CEPSP) score gain between participating Type I lesson units and participating Type II lesson units

In this section, students’ CEPSE scores were compared depending on the Type of the lesson the students participated (Type I or Type II). First students’ CEPSP score change after participating in Type I lesson was calculated by paired t-test. As Table 2 presents, after participating in Type I lesson (‘Buoyancy’ and ‘Guitar’ units), students’ CEPSP scores improved significantly. Four of five sub-categories’ score improved significantly except communication and collaboration skill; motivation (t= −2.766, *p< .05), engineering design (t= −2.567, *p< .05), engineering habits of mind (t= −3.505, *p< .05), understandings of engineering and engineers (t= −3.747, *p< .05). Overall, there was a significant difference in total score of CEPSP.

Students’ CEPSP score change after participating in Type II lesson was also calculated by paired t-test. As Table 3 presents, after participating in Type II lesson (‘Environment’ and ‘Insulation’), students’ CEPSP score was improved significantly. In sub-categories, students’ CEPSP scores improved significantly in only one sub-categories; understandings of engineering and engineers (t= −3.301, *p< .05). Overall there was no significant difference in total score of CEPSP.

This result shows that Type I lesson (engineering design that require consistent science knowledge application) is much more effective in improving students’ engineering problem solving skill compared to the Type II lesson (engineering design that is not requiring science knowledge application).

### 3.2. Comparison of students’ perceptions about Type I and Type II lesson units

#### 3.2.1. Perceptions about application of knowledge and experience

Student responded about how much they applied their knowledge and experience in each lesson unit (Table 4). For Type I lesson units about half (48.9) of the students positively responded; average 24.8% of the students responded “strongly agree” (Buoyancy (24.6%) and Guitar (25.0%)) and average 24.1% of the students responded “agree” (Buoyancy (23.1%) and Guitar (25.0%)). For Type II lesson units 37.1% of the students positively responded; average 19.5% of the students responded, “strongly agree” (Environment (21.6%) and Insulation (17.4%)) and average 17.6% of the students responded “agree” (Environment (13.5%) and Insulation (21.7%))

Figure 1 present students response rate in each lesson unit depending on their agreement level. In Type I lesson, students’ positive perception rate about knowledge and experience application is higher than Type II lesson. In other words, students perceived that they applied their knowledge and experience more in ‘Buoyancy’ and ‘Guitar’ units compared to ‘Environment’ and ‘Insulation’ units.

#### 3.2.2. Students’ perceptions about the complexity of the engineering design

Student responded about how much the engineering design in each unit was complex problem requiring logical thinking (Table 5). For Type I lesson units 34.7% of the students positively responded; average 24.1% of the students responded “strongly agree” (Buoyancy (26.2%) and Guitar (22.1%)) and average 10.6% of the students responded “agree” (Buoyancy (10.8%) and Guitar (10.3%)). For Type II lesson units 28.6% of the students positively responded; average 17.4% of the students responded “strongly agree” (Environment (20.3%) and Insulation (14.5%)) and average 11.2% of the students responded “agree” (Environment (12.2%) and Insulation (10.1%)).

Figure 2 present students response rate in each lesson unit depending on their agreement level. In Type I lesson, students’ positive perception rate about problem complexity and logical thinking requirement is higher than students’ positive perception rate in Type II lesson. In other words, more number of students participated in Type I lesson units thought engineering design they have experienced is complex problem require cognitive demands than the students participated in Type II lesson units.

Overall, after participating Type I lesson units, students’ agreement rate on their use of science knowledge and experience and the complexity of engineering design they solved is higher than participating Type II lesson units.

#### 3.2.3. Students’ perceptions about what kind of competency engineers need

Students responded about the question of asking what kind of skill or competency engineers need for effective engineering problem solving. Students’ responses before and after the engineering units was different. First, we present student’s response change after participating Type I lesson in Table 6.

As Table 6 presents, before the Type I lesson, more than half of the students perceived that science knowledge and deftness in dealing with technology and machinery is important competency and skill that engineers need to solve engineering problem. However, after the Type I lesson, ‘cooperation skill’ is the competency that the most students (34.2%) perceived that is important to solve engineering problem. They still perceived that science knowledge is important (13.1%) but the rate of the students who choose this competency was lower than before the lesson (40.6%). After the lesson students also perceived that ‘Accurate planning and design’ and ‘Efficient use of budget’ is important competency. This result shows that after the lesson, students have understood more variety skills and competency what engineer need in real world problem solving context.

Table 7 presents student’s response change after participating Type II lesson. Before the Type II lesson, the competency students perceived for engineering problem solving was very similar with the result in Table 6 (Type I lesson). 40% of the students perceived that ‘Science knowledge’ is the most important competency and 26.2% of the students perceived that ‘Deftness in dealing with technology and machinery’ is the second important competency.

Also, Students’ perceptions about engineering competency after Type II lesson is not much different with the Type I lesson participants’ that students perceived that ‘Cooperation’ skill (27.5%), Accurate planning and design (18.8%), ‘Problem solving skill (17.4%), and ‘Science knowledge (13.0%) are the most important four competencies in engineering problem solving. This result shows that both Type I lesson and Type II lesson were beneficial to the students to understand more practical and useful skills that engineers need in real world engineering problem solving context.

## 4. Conclusion and Implications

This study presents how two different types of science and engineering integrated lessons affect students’ engineering problem solving skill and their perceptions of engineering. In Type I lesson, students need to apply science knowledge they have learned to solve engineering problem whereas students in Type II lesson do not need to recall or try to understand the meaning of the concept they have learned to successfully design their engineering challenge.

The result of this study shows that after participating in Type I lesson, students’ overall CEPSP scores of engineering problem solving skill increased significantly whereas Type II lesson participated students’ CEPSP score did not increased significantly. In addition, students participated in Type I lesson agreed more on that they used their knowledge and experience in engineering design and their engineering design was complex problem solving process requiring logical thinking. This result shows that Type I lesson demands more cognitive ability and knowledge than Type II lesson. Interestingly, both Type I and Type II lesson were effective to help students perceive what competency and skill real world engineers need.

Kapur (2008) argues that purposefully designed instruction that allow learners learn from failure is critical for meaningful learning. Kapur (2006) calls this critical experience of learning from failure as ‘productive failure’. The result of this study supports Kapur (2008)’s idea that engineering design that is challenging students’ cognitive ability is more effective students to improve their engineering problem solving skill.

The result of this study implies that instruction for engineering design should consider this pedagogical approach of productive failure to make the engineering design meaningful for improving students’ problemsolving skill and knowledge. In particular, this study shows that the experience of knowledge application in engineering design is critical to improve students’ engineering problem solving skill. What kind of failure the learners experience and how they perceive the failure is critical to improving learners’ problemsolving skill (Kapur, 2008). Thus to design a meaningful integrated STEM lesson, science teachers first need to consider what knowledge of science knowledge the students should learn and apply to solve the engineering problem. More importantly, by carefully designing evaluation criteria of engineering product, teachers could make the engineering design challenging enough to student’s cognitive level.