Open access peer-reviewed chapter
Diagnostic, formative, and summative assessments, source: Adapted by authors according to Holmes-Smith [9], Earl [10], Watanabe-Crockett, Churches [7].
Abstract
Although assessment at first glance seems like a simple process in which the teacher teaches, the student learns, and then the teacher checks this students’ learned knowledge and gives the learned knowledge a certain number, in modern education all stakeholders involved are aware of the insight that not everything can be expressed in numbers. The subject of this paper’s research is the system of automatic assessment and its role in the learning process. The aim of the research conducted in the summer semester of the academic year 2020/2021 among 49 students of the undergraduate study of computing was to examine how students at the Algebra University College database-related courses assess the role of the AcpSQL assessment system in their learning process. The research result showed that students consider all possibilities and features of the AcpSQL assessment system as highly important. However, they would not generally replace manual exam correction by teachers with the automatic exam correction system, but only partially. Although students believe that the system can contribute to the assessment and self-assessment of their knowledge, they estimate that the system can only partially replace student-teacher interaction as the most important relationship in the learning process.
Keywords
- AcpSQL system
- SQL
- assessment
- automated knowledge assessment
- configurable assessment process
- higher education
1. Introduction
Assessment seems to be a simple process at first glance: the teacher teaches, the student learns, and the teacher then checks the students’ learned knowledge and gives the learned knowledge a certain number [1] (1, 2, 3, 4, 5, or in some education systems the range of grades ranges from 1 to 10). Numbers always seem to be the best or at least the most elegant solution because they look simple and unambiguous [1]. But, numbers alone cannot provide realistic, valid, and reliable assessments of students’ knowledge. However, here, the real issues connected to assessments are just beginning. The modern digital age is not modern only because of digitalization, but it has brought many other insights: that not everything can be expressed in numbers, that a qualitative approach is equal and just as important as the quantitative one, that people, students in this context, are unique, so the approach cannot be all-the-same-of-a-kind, but an adjusted approach and ‘unique assessment processes’ to ‘one-of-a-kind’ students [2].
In the following chapter, the assessment process will be presented with a special emphasis on the evaluation purpose, addressing the importance of the entire context, which includes the assessment objectives, the methods used, the quality of the assessment, and the interpretation of the assessment results. After the theoretical insights, research conducted on a sample of Algebra University College students who experimentally used the Automated Programming Assessment System (APAS) called AcpSQL will be presented. Also, the results of an anonymous survey conducted on the same sample with questions that differentiate assessment FOR learning from assessment AS learning and assessment of learning will be presented. The survey also included general questions about APAS features, benefits, usability, and overall benefits and usefulness.
Advertisement
2. Knowledge assessment and assessment purpose
Assessment, in its simplest form, should represent a systematic process in which the teacher collects data, analyzes it, and interprets it to determine the extent to which students have mastered educational goals. It is most often carried out through:
observation and related techniques (e.g., oral examination), which have a qualitative character (e.g., the student is getting better in some field),
measurement, which refers to the process of determining a numerical measure of someone’s achievement or trait and has a quantitative procedure, not a qualitative one (student solved 15 out of 20 tasks), and
evaluation – giving value to assessed knowledge – assessment [1].
Considering that measurement of knowledge belongs to the field of indirect measurements (subject of measurement and units of measurement are not the same type), research in this area shows that the source of error can be in:
the subject of measurement – knowledge is not assessed directly, but through students’ answers,
measuring instrument, and
measurement techniques, where the teacher most often appears in the role of a measurer and, at the same time, in the role of a measuring instrument [1].
To reduce the influence of subjective factors of teachers on assessment, there is a tendency to develop objective tests, which have standardized procedures and assume that the conditions are defined in advance, and that each student has the same instructions.
Assessment can have many different purposes:
to provide feedback to students about their current accomplishments,
to provide feedback about students’ progress,
to motivate students’ (to learn more),
to provide pedagogical feedback to students, which allows them to compare their success with expected standards, to use detailed feedback to correct and improve their work, and to understand more clearly the requirements of a particular activity [3],
for selection (e.g., passing the exam, passing the course, enrolling in a higher year, and enrolling in a higher level of education),
to pinpoint weaknesses in students’ knowledge and to aid students in their learning [2],
Formative assessment is an assessment AS learning as opposed to an assessment of learning, as explained by Wabisabi Learning [2]. Its focus is on looking backward and forward, as well as feeding back and forward as learning happens, and its purpose is to establish where students want to be and help them get there.
Assessment FOR learning is a type of formative assessment; its focus is on engaging students in classroom assessment in support of their learning and informing teachers about what to do next to help students progress, and it is for improvement, not assessment for accountability as can be the case with summative assessments [5].
Assessment OF learning is a summative assessment used primarily to compare students and report progress [6], most often conducted through unit tests.
Summary of the brief snapshot of the assessments that are most common in education and their objectives and outcomes [7], the required range of assessment practices to be used [8, 9], and planning assessment AS, FOR, and OF learning [10] is presented in Table 1.
| Assessment Type | Diagnostic | Diagnostic and formative | Summative |
|---|---|---|---|
| Allows | Looking back Preparing forward | Looking back and preparing forward Feeding back and feeding forward | Feeding back Providing a snapshot |
| Assessment Objectives and Outcomes | Assessment FOR Learning | Assessment AS Learning | Assessment OF Learning |
| Requires these assessment practices | Ascertaining students’ prior knowledge, perceptions, and misconceptions; monitoring students learning progress; and informing teaching practice and curriculum planning to support students’ future learning and understanding | Focusing on constructive feedback from the teacher and on developing students’ capacity to self-assess and reflect on their learning to improve their future learning and understanding | Making judgments about what the student has learned concerning the teaching and learning goals should be comprehensive and reflect the learning growth over the period being assessed |
| Why assess? | to enable a teacher to determine the next steps in advancing student learning | To guide and provide opportunities for each student to monitor and critically reflect on his or her learning and identify the next steps | To certify and inform parents or others of student’s proficiency concerning curriculum learning outcomes |
| Assess What? | Each student’s progress and learning needs concerning the curricular outcomes | Each student’s thinking about his or her learning, what strategies he or she uses to support or challenge that learning, and the mechanisms he or she uses to adjust and advance his or her learning | The extent to which students can apply the key concepts, knowledge, skills, and attitudes related to the curriculum outcomes |
| What methods? | A range of methods in different modes that make students’ skills and understanding visible | A range of methods in different modes that elicit students’ learning and metacognitive processes | A range of methods in different modes that assess both product and process |
| Ensuring quality |
|
|
|
| Using the information |
|
|
|
Table 1.
Advertisement
3. An example of good practice: AcpSQL system assessment at the algebra university college database-related courses
At Algebra University College, teachers responsible for database-related courses and authors of this chapter developed an automatic, configurable, and partial assessment system for students’ SQL queries [11], abbreviated AcpSQL System. The proposed automated assessment system allows an assessment system with rules for subtracting points defined by teachers, which should reflect learning outcomes already defined in the course. It is the teachers’ obligation to relate them as precisely as possible because the feedback provided by the assessment system is of more use for students, and grading is more detailed, thus more transparent for the student. At the database SQL courses at Algebra University College, for which this assessment system is primarily developed, each learning outcome is divided into minimal and desired learning outcomes, and each of those is related to practical tasks for students to develop during the exam. Thus, the assessment system is strictly related to minimal and desired learning outcomes, helping students know what part of the learning outcomes they passed and what part they could do better (described by the system in detail).
In addition to helping students, the assessment system AcpSQL System at Algebra University College has been developed as help for teachers because the number of assignments that teachers have to assess during exam terms is measured in thousands [12]. The usability of the assessment system can also be extended as it is used FOR learning to better understand students’ prior knowledge, to follow students’ progress in learning and achieving learning objectives, and to make evaluation criteria more transparent. Students get a detailed description of their lapses, so they can better prepare later on, and teachers, unburdened by manual assessment and later insights, can think over learning objectives and methods used.
The assessment system is designed and developed to help teachers understand students’ prior knowledge regarding predefined learning objectives. If the assessment system addresses students’ knowledge gaps, teachers are the ones who should consider that cognition when choosing the mix of learning methods that they would use to best match someone’s needs. Teachers can then fine-tune the approach and make the additional effort, which results in students’ better understanding and, consequently, greater benefit from the whole course.
In teacher experiences at the Algebra University College, at the database-related courses (Introduction to Databases, obligatory for Software Engineering students and System Engineering students and Database Development, obligatory only for Software Engineering students), it stands out that students themselves are not aware of their knowledge gaps, since they often repeat their lapses at following exam terms. In practice, if someone has prior knowledge on one topic, he or she cannot estimate how many and how deep are the knowledge gaps regarding that same topic. From that point of view, the assessment system also helps discover those gaps consistently and objectively.
Another aspect of using the assessment system is to understand if there is students’ progress during the semester as planned. If not, teachers can use that information and think over their methods and learning objectives. In case of only occasional and not always the same gaps, it should not result in learning objective or methods changes because that is not unexpected during the learning period, but when some pattern appears, it is indicative and valuable for teachers. The initial assessments, before the learning process begins, and those assessments during the semester are comparable, since the same assessment system and the same grading rules created by the same teachers are used, making the outputs valuable for evaluation of student progress during the semester.
AcpSQL assessment system uses the model of correct answers created by teachers to compare the result sets with student answers, but the teacher also must define clear grading rules in advance, so that the system can give partial points for student answers, which can be executed, but with one or more imperfections in their solution [12]. Since not many systems have this possibility [13, 14, 15, 16, 17, 18, 19, 20, 21, 22], it was one of the main purposes to achieve this AcpSQL assessment system model. That way, students can be aware of the criteria before developing their solution, and they get the feedback with a detailed description so they can match where they made lapses and how many percentages of points they did not achieve because of them. This information helps them overcome their knowledge gaps. In addition, students can get that description almost immediately after submitting their solution, rather than waiting a few days to get it when teachers award student solutions manually. This model also provides detailed information for the student’s parents or guardians of a student, making them informed as detailed as possible of the particular exam term outcomes, which a student took. This can also be an important factor in student efficacy.
Another advantage of the proposed AcpSQL assessment system is the fact that students can use it in addition to usual teaching. That way, it makes a complement method for them that is not time-limited, and which helps that their success depends only on their motivation. If used in that fashion, the progress is strongly related to the student’s efforts, annulling possible teacher or teacher’s assistant occasional unavailability. When students get the habit of using the assessment system, it will supplement the usual classes, since students can practice and get immediate feedback more often and faster than in the usual classes. Also, they will be in a position to ask more concrete questions, targeting specific issues they have, getting specific answers from teachers, and making their learning process more efficient. It means that the system can be used as an assessment OF learning.
If the usage of an assessment system is significant, the advanced way of using it is to collect and analyze the data of each student. It is possible to analyze the progress, discover the fields with insufficient progress, and help students and teachers to invest extra effort to overcome those. An analysis can also discover learning outcomes that are defined maybe as too easy, and learning outcomes that are traditionally too difficult for a significant number of students.
Regular classes of database-related courses at the Algebra University College are designed to lead students to the possibility of easily coping with practical tasks later. Each learning outcome of those courses covers the most often tasks students will face at their future jobs. Ideally, students who finish those courses will be able to start working immediately or with minimal adjustment. However, this could be problematic for some students if they have learned only theory on predefined learning objects. Their knowledge and learned skills would be greater if they could use combined methods in learning and practicing what they have learned on practical tasks. With consistent and almost immediate feedback, combining learning methods by using the assessment system as a supplement to traditional teaching would enhance learning achievements and retention [23].
In general, it is expected that the assessment system is consistent, creating surroundings in which students are confident to use it. It means that each time a student makes the same lapse in his or her solution, the system reacts the same way, subtracting the same number of points. The additional advantage is that the rules that the system uses when deciding how many points it will subtract are known in advance. Teachers can define them before students take the exam, and it is happening in the system, similarly as teachers think of it before manually grading student solutions. The important difference is that when using the assessment system, teachers must write those rules in a configuration file before grading, so it is clear what the important parts of student solutions are and what percentage of points will be subtracted if some of those rules are broken. The additional advantage is that students get descriptive feedback based on those rules after taking an exam, knowing exactly where they made mistakes and how many points were subtracted according to those lapses, minimizing their need for insight organized by teachers, by personal contact, email, or any other communication channel.
The assessment system can also be useful since it provides the ability to practice practical tasks early during learning. Also, it is students’ valuable first experience with practice tasks, making them more prepared for real-life projects.
In addition to everything mentioned above, the system can also help in recognizing potential similar problematic situations in an early phase, yet during the education process. That cognition could strengthen students’ chances to cover all existing gaps and to start participating in their first job much easier.
An example of AcpSQL assessment system feedback given at the real exam term at Algebra University College is shown below. It shows important information for each SQL clause (in this case, WHERE, HAVING, SELECT, and ORDER BY):
First, the reason why it is wrong (in this example, wrong rows filtered, missing column, and the wrong column in order by clause),
coming up with the percentage of maximum points remain (also clear the subtracted percentage),
and at the end, a final mark in points and percentage [12]:
starting to mark: Jane Doe.sql
where: wrong rows filtered, remaining 90.0%
having: wrong number of rows, remaining 90.0%
select: missing column city.countryid remaining 86.0%
orderby: missing column city.countryid, remaining 81.0%
orderby: column 1 should be countryid, remaining 76.0%
Jane Doe.sql marking complete, final mark: 3.04/4.00 (76.0%)
When AcpSQL assessment system was developed, it worked with only fundamental SQL clauses like SELECT, FROM, WHERE, and ORDER BY [11]. In its second version, it also covers clauses GROUP BY, HAVING, and all types of JOIN clauses [12].
Learning outcomes of the database-related courses at the Algebra University College cover not only SQL clauses mentioned above but also more complex tasks, which students should resolve using the combination of more than one SQL statement, not limited only to SELECT, but also INSERT, UPDATE and DELETE statements, in particular, order [11]. However, the AcpSQL assessment system development is still ongoing to make its use even more valuable to students in the future.
Automatic grading systems are among the innovative methods, not only of grading but also of learning and teaching, not only by the teacher (mentor/teacher-student) but also by the individual student (student-content), mastering new teaching contents with fellow students (learning: student–student) as well as mastering new teaching contents through technological media, in various ways (student-online interactive information and communication system that enables self-assessment and self-evaluation or/and student-online). Accordingly, for this example of good practice to evolve, a survey was conducted on students’ attitudes toward assessment methods and the possibilities and advantages of an automatic assessment system.
Advertisement
4. The research design and sample
The subject of research in this chapter is the system of automatic assessment and its role in the learning process. This research aims to examine how students at the Algebra University College database-related courses assess the role of the AcpSQL assessment system in their learning process and, more accurately, to explore how students assess the impacts of automatic assessment system use FOR learning, AS learning, and OF learning.
This subject of research is generally actual, relevant, and important for a wide population of students, but also for teachers. In addition, the results of this research will be used for possible refinement of the system to check the general attitudes of students toward these types of assessment, i.e., which features of the automatic assessment system are considered advantageous by students.
The research aim of this study was to identify the factors that influence the preferred form of assessment at the Algebra University College database-related courses. To achieve the goal of the research, the following research questions have been specified:
Which features of assessment affect students’ preferred form of assessment at the Algebra University College database-related courses?
Which features of the automatic assessment system affect the students’ preferred form of assessment at the Algebra University College database-related courses?
Which form of assessment do students prefer at the Algebra University College database-related courses?
What types of interaction when learning (mastering new knowledge and skills) are preferred by students at the Algebra University College database-related courses?
To answer these research questions, a specially constructed instrument – a questionnaire was designed. The survey was conducted during the summer semester of the 2020/2021 academic year, using a specially designed questionnaire in the Google Forms tool, which was distributed to students via email and/or QR code. Students’ participation was voluntary and anonymous. The questionnaire consisted of 13 closed-ended questions, 5 of which were about their general demographics and 8 which referred to the preferred form of evaluation and (self) evaluation. To ensure a clear understanding of the purpose of this research before answering the questions, it was specified at the beginning of the questionnaire that the research seeks to determine what type of evaluation students prefer.
The research sample consisted of 49 students of the undergraduate study program (100%) of computing who participated in the conducted survey at the database SQL courses of the Algebra University College Zagreb. A sample was pertinent. The structure of the sample was as follows: 73.5% of students were students of Software Engineering, 10.2% of System Engineering, and 16.3% of students enrolled in the study of Multimedia Computing. The overall ratio of student participants according to gender was 77.6% male student population and 22.4% female population. The male gender was significantly more represented among the students. The structure of the sample of students in the preliminary research was dominated by students who have the status of full-time students (89.8%), while the number of employed students was far smaller (10.2%), as well as first-year students (89.8%), second-year students (only 10.2%) and third-year students, none of whom participated in this research. The demographic statistics of the total sample (N = 49) are shown in Table 2.
| Demographics | Category | Percentage |
|---|---|---|
| Level of study | Undergraduate | 100% |
| Graduate | 0% | |
| Field of computing study | Software Engineering | 73.5% |
| System engineering | 10.2% | |
| Multimedia computing | 16.3% | |
| Year of study | First-year | 89.8% |
| Second-year | 10.2% | |
| Third-year | 0% | |
| Student status | Full-time student | 89.8% |
| Part-time student | 10.2% | |
| Gender | Men | 77.6% |
| Women | 22.4% |
Table 2.
Distribution of respondents by study level, computing field, study year, student status, and gender (N = 49). Source: Questionnaire and authors’ analysis.
Advertisement
5. The research results
An anonymous survey was conducted via a designed structured questionnaire at the compulsory lectures of undergraduate professional students of Applied Computing (fields Software and System Engineering). Students voluntarily filled out an anonymous survey questionnaire, and no ambiguities were reported in the questionnaire, from which it can be concluded that the questions were unambiguous, and the particles were clear. The data were processed by quantitative statistical analysis of preisolated variables and a description of the conditions and the establishment of cause-and-effect relationships between the variables.
In the construction of the measuring instrument, the Holmes-Smith [9] research, according to the Department of Education and Training [8], Earl [10], Watanabe-Crockett, Churches [7], was used. Using their concluding categories, a measuring instrument was created that refers to assessment types (FOR, AS, and OF learning) and their features’ importance. With the help of adapted particles from the Fabijanić, Đambić, Fulanović research [11] according to Al-Salmi [14] and Đambić, Fabijanić, Lokas Ćošković [12], the questions related to automatic assessment system features were created. In the categories related to the advantages and disadvantages of the assessment system, the adaptation of certain particles from the research Babić [24] according to Grosseck [25], Mabić [26], Raut, and Patil [27] was made, while particles that refer to preferred interaction when learning (mastering new knowledge and skills) was taken over from Babić, Vunarić, and Lokas Ćošković research [28].
Respondents who were surveyed used the AcpSQL assessment system at the Algebra University College database-related courses for at least one academic year (first-year students) or two years (second-year students). In this sense, it is assumed that the answers represent general opinions based on past or present own experiences of using the system. Below is a tabular presentation of the survey results expressed in percentages. To realize the set research goal, an overview of the results according to the specified research questions is the following:
5.1 What features of assessment affect students’ preferred form of assessment at the algebra university college database-related courses?
For examining the assessment features that influence the preferred form of assessment by students at the Algebra University College database-related courses, the first three questions of the closed questionnaire asked respondents to grade each of the listed assessment features FOR, AS, and OF learning with a degree of importance from 1 to 5. Responses were predetermined and adjusted according to Holmes-Smith [9] and the Department of Education and Training [8], as presented earlier in this chapter. Table 3 shows the distribution of responses about the importance of features (variables) of assessment for learning, of assessment as learning, and of assessment as learning according to 5 degrees of the Likert scale, for Algebra University College students, N = 49.
| Assessment types and features (variables) importance | Not at all important | Slightly important | Moderately important | Very important | Extremely important |
|---|---|---|---|---|---|
| Determining student’s prior knowledge and their perceptions of the course | 2% | 4.1% | 22.5% | 40.8% | 30.6% |
| Enabling tracking of learning progress | 0% | 0% | 4.1% | 44.9% | 51% |
| Planning future steps in learning | 0% | 0% | 0% | 61.2% | 38.8% |
| Getting continuous feedback from teachers | 0% | 0% | 0% | 46.9% | 53.1% |
| Developing students’ self-assessment skills | 0% | 4.1% | 2% | 44.9% | 49% |
| Students’ reflection on their learning to improve their future learning and understanding | 2% | 0% | 4.1% | 34.7% | 59.2% |
| Getting teachers ‘judgments about what they have learned concerning teaching and learning goals | 0% | 2% | 6.2% | 44.9% | 46.9% |
| Comprehensive evaluation | 0% | 0% | 12.2% | 44.9% | 42.9% |
| Reflecting learning growth over the period for which it is estimated | 0% | 0% | 6.2% | 46.9% | 46.9% |
Table 3.
Regarding the subdomain assessment FOR learning features, the results show that most students, 100% of them, believe that ‘planning future steps in learning is important’, for 61.2% of them, it is very important, and it is extremely important for 38.8%. None of the students think it does not matter. 71.4% of students consider ‘determining students’ prior knowledge and their perceptions of the course’ very or extremely important, and only 2% not important, while for 22.5% of them, it is moderately important. Likewise, ‘enabling tracking of learning progress’ is very or extremely important for a total of 95.9% of the respondents, and none of the students chose the option that this feature is not important.
When it comes to subdomain assessment AS learning features, the results show that most students, 100% of them, think that ‘getting continuous feedback from teachers’ is very important (46.2%) or extremely important (53.1%). None of the students think it does not matter. While a total of 93.9% of students consider ‘developing students’ self-assessment skills’ very or extremely important, and for 2% it is moderately important; for 4.1% of respondents, it is slightly important. None of the students selected the option that this feature is not important. The feature ‘students’ reflection on their learning to improve their future learning and understanding’ is very or extremely important for a total of 93.9% of the respondents; for 4.1% of students, it is moderately important, but 2% of students think it is not important at all.
As for the features of subdomain assessment OF learning, results show that the feature ‘reflecting learning growth over the period for which it is estimated’ is considered extremely important by 46.9% of students, very important for 46.9%, 93.8% in total. Only 6.2% rated it moderately important, while it is not considered unimportant by any of the students. ‘Getting teachers’ judgments about what they have learned concerning teaching and learning goals’ is extremely important for 46.9% of students, very important for 44.9%, 91.8% in total. For 6.2%, it is moderately important, and for 2%, it is slightly important. None of the students consider this unimportant. Regarding the ‘comprehensive evaluation’, a total of 87.8% of respondents consider it very important (44.9%), extremely important (42.9%), and 12.2% moderately important. None of the students consider this feature unimportant.
5.2 Which features of the automatic assessment system affect the students’ preferred form of assessment at the algebra university college database-related courses?
To examine which automatic assessment system features students consider most important, students were asked, based on their use of the AcpSQL assessment system, to choose the degree of importance of 1–5 for 9 most important features of the system according to Fabijanić, Đambić, Fulanović research [11], Al-Salmi [14] and Đambić, Fabijanić, Lokas Ćošković [12].
Table 4 shows the distribution of responses about the importance of automatic assessment system features at database-related courses of Algebra University College students, N = 49. The results show that as the most important feature of the automatic assessment system students evaluate ‘quick and concrete comment on the shortcomings of the solution’ (total 98% of them consider it very and extremely important), then ‘predefined and student-communicated criteria’ (total 95.9% of them consider it very and extremely important), followed by the feature ‘possibility to adjust the criteria and their shares in the points won for each task’ (total 91.8% of them consider it very and extremely important)), ‘possibility of repeated use during learning’ (total 89.8% of them consider it very and extremely important). All the offered features of the system are considered very important by most students, so they follow the ‘interactive approach especially useful during online education’ (total 87.8% of them consider it very and extremely important), ‘consistency of correction’ (total 85.7% of them consider it very and extremely important), ‘availability during and outside classes’ (total 77.6% of them consider it very and extremely important), a ‘large number of possible evaluation criteria’ (total 67.35% of them consider it to be very and extremely important), and ‘speed of exam correction and points’ (total 65.3% of them consider it very and extremely important). Of all the features offered, the feature ‘speed of exam correction and points’ has the least importance for the respondents, so 2% of students marked it as not at all important and 6.2% as slightly important.
| Automatic assessment system features (variables) importance | Not at all important | Slightly important | Moderately important | Very important | Extremely important |
|---|---|---|---|---|---|
| Quick and concrete comment on the shortcomings of the solution | 0% | 2% | 0% | 47% | 51% |
| Predefined and student-communicated criteria | 0% | 0% | 4.1% | 34.7% | 61.2% |
| Possibility to adjust the criteria and their shares in the points won for each task | 0% | 0% | 8.2% | 46.9% | 44.9% |
| Possibility of repeated use during learning | 0% | 0% | 10.2% | 46.9% | 42.9% |
| An interactive approach especially useful during online education | 0% | 0% | 12.2% | 42.9% | 44.9% |
| Consistency of correction | 0% | 4.1% | 10.2% | 26.5% | 59.2% |
| Availability during and outside classes | 0% | 2% | 20.4% | 42.9% | 34.7% |
| A large number of possible evaluation criteria | 0% | 4.1% | 28.55% | 38.8% | 28.55% |
| Speed of exam correction and points | 2% | 6.2% | 26.5% | 40.8% | 24.5% |
Table 4.
Distribution of responses about the importance of the automatic evaluation system features, N = 49. Source: Questionnaire adapted according to Al-Salma (2019) and Đambić, Fabijanić, Lokas Ćošković [12] and authors’ analysis.
To examine in more detail which features of the system students consider being advantages, in the next question, students were able to mark all the features they consider to be an advantage. Twenty-four features were offered that were adapted according to Babić [24], Grosseck [25], Mabić [26], Raut, Patil [27], and there was a possibility to enter their answer. As shown in Table 5, as the biggest advantage of the automatic assessment system, as many as 75.5% of respondents singled out ‘faster feedback’. Also, more than half of the students singled out ‘easier and faster access to results’ (67.3%) and ‘reduced time and costs required for evaluation’ (65.3%) as the most significant advantages. In the top five advantages, they also singled out ‘reduced possibility of teachers’ mistakes’ and ‘possibility of self-assessment and self-evaluation (42.9%). The least significant advantages, in their opinion, are ‘exam experience is more interactive and interesting’ (4.1%), ‘ability to control access through user authentication’ (8.2%), and ‘flexibility in terms of technology selection’ (10.2%). Although the option to introduce some additional benefits existed, students did not enter any of their suggestions. All advantages, as well as the percentage of responses related to them, are shown in more detail in Table 5.
| Automatic assessment system advantages | N | % |
|---|---|---|
| Faster feedback | 37 | 75.5% |
| Easier and faster access to results | 33 | 67.3% |
| Reduced time and costs required for evaluation | 32 | 65.3% |
| Reduced possibility of teachers’ mistakes | 21 | 42.9% |
| Possibility of self-assessment and self-evaluation | 21 | 42.9% |
| Exams can be taken in digital form | 16 | 32.7% |
| Possible access from different locations | 16 | 32.7% |
| Focus on innovation in learning, not on the technology itself | 14 | 28.6% |
| Ability to test existing teaching models | 12 | 24.5% |
| Ability to test existing grading models | 11 | 22.4% |
| Acquisition of IT education | 10 | 20.4% |
| Possibility of integration of various technologies in future learning and teaching activities | 10 | 20.4% |
| Easier and faster test creation | 9 | 18.4% |
| Possibility of integration of various technologies in assessment activities | 9 | 18.4% |
| Increased ways of grading due to the diversity of new technologies | 8 | 16.3% |
| Reliability in continuous use over a long period | 8 | 16.3% |
| Low level of complexity of use (minimum skills required) | 7 | 14.3% |
| Increased correction reliability | 7 | 14.3% |
| Reduced possibility of teachers’ ‘bias’ | 7 | 14.3% |
| Ability to create digital content | 6 | 12.2% |
| Compatibility with different areas of education | 6 | 12.2% |
| Flexibility in terms of technology selection | 5 | 10.2% |
| Ability to control access through user authentication | 4 | 8.2% |
| Exam experience is more interactive and interesting | 2 | 4.1% |
| Other | 0 | 0% |
To examine in more detail the attitudes of students toward the automatic assessment system self-assessment advantages, students were asked to indicate which features they considered as advantages of the automatic assessment system for students’ self-evaluation. The option of multiple choice was enabled of 8 offered advantages, as well as the possibility to enter their answers. Most of the respondents singled out ‘direct interaction with the system and possible subsequent verification with teachers’ as the biggest advantage of the automatic assessment system for self-assessment/self-evaluation purposes (81.6%). More than half of the survey participants put ‘time saving (for transport, breaks or “time holes” between exams)’ (61.2%) and ‘direct interaction with the system and possible comparison with common mistakes of other students’ (55.1%) as top 3 advantages. The least prominent advantage was ‘no direct interaction with teachers is required’. Respondents did not enter any additional advantages. All advantages, as well as the percentage of responses related to them, are shown in more detail in Table 6 that follows.
| Automatic assessment system self-assessment/self-evaluation advantages | N | % |
|---|---|---|
| Direct interaction with the system + possible subsequent verification with teachers | 40 | 81.6% |
| Timesaving (for transport, breaks, or ‘time holes’ between exams) | 30 | 61.2% |
| Direct interaction with the system + possible comparison with common mistakes of other students | 27 | 55.1% |
| Easier adoption of the material due to the direct experience of evaluation | 15 | 30.6% |
| Organized exact maintenance time | 12 | 24.5% |
| Organized exact location of maintenance | 8 | 16.3% |
| Ability to work in a team | 7 | 14.3% |
| No direct interaction with teachers is required | 6 | 12.2% |
| Other | 0 | 0% |
Table 6.
Distribution of responses about automatic evaluation system self-assessment/self-evaluation advantages, N = 49. Source: Questionnaire and authors’ analysis.
5.3 Which form of assessment do students prefer at the algebra university college database-related courses?
To put the automatic assessment system in context with other types of evaluation at the Algebra University College database-related courses, students were asked to choose a preferred exam assessment type in a closed-ended question, that is, if they would replace manual exam correction (by teachers) with automatic exam correction systems. Most students would replace manual exam correction (by teachers) with automatic exam correction systems but just partially (69.4%), and less than 5% would do so completely (only 4.1%). Less than one-fifth of respondents would not do so at all (18.4%), while 8.2% do not have an opinion on the subject, as shown in Table 7.
| The preferred exam assessment system | N | % |
|---|---|---|
| Would replace manual exam correction (by teachers) with automatic exam correction systems completely | 2 | 4.1% |
| Would replace manual exam correction (by teachers) with automatic exam correction systems partially | 34 | 69.4% |
| Would not replace at all manual exam correction (by teachers) with automatic exam correction systems | 9 | 18.4% |
| No opinion on the subject | 4 | 8.2% |
Table 7.
Distribution of responses about preferred exam assessment system, N = 49. Source: Questionnaire and authors’ analysis.
5.4 What types of interaction when learning (mastering new knowledge and skills) are preferred by students at the algebra university college database-related courses?
The final question in this study concerns the preferred types of interaction when learning, i.e., mastering new knowledge and skills by students at the Algebra University College database-related courses. The question was closed-ended, and the students were offered adapted options according to Babić, Vunarić, Lokas Ćošković research [28].
Table 8 shows that the largest number of respondents, i.e., more than half of the respondents, prefer ‘direct face-to-face teaching by the teacher/mentor (learning: mentor/teacher-student)’ (55.1%). Results also show that the second preferred type is ‘individual mastering of the content of the lecture (learning: student-content)’, which is preferred by about a fifth of the respondents (20.4%). An equal percentage of students, about 10.2% prefer ‘individual mastering of the content of the lecture (learning: student-content)’, as well as ‘mastering new teaching contents through technological media (learning: student-online)’. Only 4.1% chose ‘mastering new teaching contents through technological media (learning: student-online interactive information and communication system that enables self-assessment and self-evaluation)’ as the preferred style of interaction when learning.
| Preferred interaction when learning (mastering new knowledge and skills) | N | % |
|---|---|---|
| Individual mastering of the content of the lecture (learning: student-content) | 10 | 20.4% |
| Mastering new teaching contents with fellow students (learning: student–student) | 5 | 10.2% |
| Direct face-to-face teaching by the teacher/mentor (learning: mentor/teacher–student) | 27 | 55.1% |
| Mastering new teaching contents through technological media (learning: student-online interactive information and communication system that enables self-assessment and self-evaluation) | 2 | 4.1% |
| Mastering new teaching contents through technological media (learning: student-online) | 5 | 10.2% |
Table 8.
Distribution of responses about preferred interaction when learning (mastering new knowledge and skills), N = 49. Source: Questionnaire adapted according to Babić, Vunarić, Lokas Ćošković [28], and authors’ analysis.
Advertisement
6. Discussion of research results
A discussion of the results presented by research questions and subdomains is as follows:
Assessment FOR learning, assessment AS learning, and assessment OF learning are highly important
The results of this research show, as far as assessment FOR learning is concerned, that all three features: ‘planning future steps in learning’ (100%), ‘enabling tracking of learning progress’ (95.9%), and ‘determining student’s prior knowledge and their perceptions of the course ‘(71.4%) are very or extremely important according to students. Regarding assessment AS learning features, the results showed that ‘getting continuous feedback from teachers’ was also rated by 100% as strong or extremely important, and in an equally high percentage, they also rated ‘developing students’ self-assessment skills’ (93.9%). The feature ‘students’ reflection on their learning to improve their future learning and understanding ‘is also highly rated by 93.9%. When it comes to assessment OF learning, the results showed that students consider ‘reflecting learning growth over the period for which it is estimated’ (93.8%) also very or extremely important, and it is similar to the ‘getting teachers’ judgments about what they have learned concerning teaching and learning goals’ (91.8%) and ‘comprehensive evaluation’ (87.8%). The results concerning all 3 learning subdomains and their features show that students at the Algebra University College database-related courses confirm that all the features defined by the Holmes-Smith [9] and the Department of Education and Training [8] research they consider very or extremely important.
All automatic assessment system features are highly important
The research results show that students consider that the most important feature of the automatic assessment system is ‘quick and concrete comment on the shortcomings of the solution’ (98%), followed by ‘predefined and student-communicated criteria’ (95.9%), ‘possibility to adjust the criteria and their shares in the points won for each task’ (91.8%), and ‘possibility of repeated use during learning’ (89.8%). It is significant to notice that all features of the system are considered very important by most students; ‘interactive approach especially useful during online education’ (87.8%), ‘consistency of correction’ (85.7%), ‘availability during and outside classes’ (77.6%), ‘a large number of possible evaluation criteria’ (67.35%), and of all features offered ‘speed of exam correction and points’ (65.3%) has the least importance for the respondents, but still most students also consider this feature highly important.
The most significant automatic assessment system advantages: faster feedback, easier and faster access to results, and reduced time and costs required for evaluation
Of 24 possible advantages adapted according to Babić [24], Grosseck [25], Mabić [26], Raut, Patil [27] and offered to students for evaluation, students highlighted ‘faster feedback’ (75.5%), ‘easier and faster access to results’ (67.3%) and ‘reduced time and costs required for evaluation’ (65.3%) as the most significant advantages. Additionally, ‘reduced possibility of teachers’ mistakes’ and ‘possibility of self-assessment and self-evaluation’ (42.9%) are singled out in the top 5 most significant automatic assessment system advantages. At the same time, the least significant advantages are ‘flexibility in terms of technology selection’ (10.2%), ‘ability to control access through user authentication’ (8.2%), and the least important in students’ opinion: ‘exam experience is more interactive and interesting’ (4.1%).
The most significant automatic assessment system self-assessment advantages: direct interaction with the system and possible subsequent verification with teachers
Of 8 offered advantages, students’ opinion is that ‘direct interaction with the system and possible subsequent verification with teachers’ is the most significant advantage of the automatic assessment system for self-assessment purposes (81.6%), followed by ‘timesaving (for transport, breaks, or “time holes” between exams)’ (61.2 %) and ‘direct interaction with the system and possible comparison with common mistakes of other students’ (55.1 %). The least prominent advantage is ‘no direct interaction with teachers is required’.
Preferred exam assessment system: students would only partially replace manual exam correction (by teachers) with automatic exam correction systems
In context with other types of evaluation at the Algebra University College database-related courses, students would replace manual exam correction (by teachers) with automatic exam correction systems, but only partially (69.4%). Only 4.1% would do so completely. Less than one-fifth of respondents would not do so at all (18.4%), while 8.2% do not have an opinion on the subject.
Direct face-to-face teaching by the teacher/mentor (learning: mentor/teacher–student) is still the preferred interaction when learning (mastering new knowledge and skills)
The research results showed that the majority of students still prefer ‘direct face to face teaching by the teacher/mentor (learning: mentor/teacher-student)’ (55.1%) when learning (mastering new knowledge and skills). But it is significant that the second preferred type is ‘individual mastering of the content of the lecture (learning: student-content)’ (20.4%). About 10% of students equally prefer ‘individual mastering of the content of the lecture (learning: student-content)’ and ‘mastering new teaching contents through technological media (learning: student-online)’. It is significant to notice that only 4.1% prefer ‘mastering new teaching contents through technological media (learning: student-online interactive information and communication system that enables self-assessment and self-evaluation)’.
Advertisement
7. Conclusion
This chapter provides an insight into the increasingly popular use of automatic assessment systems and students’ estimates and preferences of their use effects on students’ learning experiences. The aim of this research was to examine and identify the factors that influence the preferred form of assessment at the Algebra University College database-related courses. Although the chapter needs methodological corrections, it provides insight into the existence of a certain connection between the automatic assessment system and the learning process.
The research showed that in the first research question about the features of assessment that affect students’ preferred form of assessment at the Algebra University College database-related courses, students estimate that all types of assessment, i.e., assessment FOR learning, assessment AS learning and assessment OF learning and their features are highly important, where all 9 selected variables are considered very or extremely important for a minimum of 70% up to 100% of the students. More specifically, as far as the second research question is concerned, i.e., when it comes to features of the automatic assessment system that affect the student’s preferred form of assessment at the Algebra University College database-related courses, as the most important feature of the automatic assessment system about 90% of surveyed students pointed out ‘quick and concrete comment on the shortcomings of the solution’, ‘predefined and student-communicated criteria’, ‘possibility to adjust the criteria and their shares in the points won for each task’, and ‘possibility of repeated use during learning’. But, most students consider all mentioned features of the AcpSQL assessment system, including ‘interactive approach especially useful during online education’, ‘consistency of correction’, ‘availability during and outside classes’, ‘a large number of possible evaluation criteria’, and ‘speed of exam correction and points’ as highly important.
It is interesting to note that, according to the students, speed, and feedback are at the forefront of all possible automatic assessment systems’ benefits. So, the results showed that students at the Algebra University College database-related courses as the most significant automatic assessment system advantages highlight faster feedback, easier and faster access to results, and reduced time and costs required for evaluation, whereas the most significant automatic assessment system self-assessment advantages they highlighted direct interaction with the system and possible subsequent verification with teachers. As for the third research question about the preferred exam assessment system, research results showed that students, however, would not generally leave the assessment to an automatic system, but would only partially replace manual exam correction (by teachers) with automatic exam correction systems. As for the fourth, also the last research question, concerning types of interaction when learning in terms of mastering new knowledge and skills, the research results showed that the majority of students at the Algebra University College database-related courses still prefer direct face-to-face teaching by the teacher/mentor and learning process that includes mentor/teacher and student interaction. However, as a very interesting fact, it is pointed out that only 4.1% of students prefer mastering new teaching content through technological media and learning process that includes student-online interactive information and communication system that enables self-assessment, which is even more interesting and significant data if we consider that the data came from digital generation students and more importantly, computing students.
To conclude, students consider the features of the AcpSQL System to be highly important but estimate that the system can only partially replace student-teacher interaction as the most important relationship in the learning process.
Advertisement
8. Limiting elements of the research conducted and recommendations for future research
Before concluding the chapter, it is important to emphasize the difficulties encountered during the implementation of the research and, ultimately, its limiting elements. First, a pertinent and very small sample was used – undergraduate first and second-year computing students, the ones found on the day of the research in the class. The way the survey was distributed (at the Algebra University College database-related courses) was limiting for sampling. The respondents’ number should be much higher in future research so that the target population can be inferred with greater precision based on the sample. Second, for future research of automatic assessment systems, it would be useful to refine the AcpSQL assessment system so that it can be used for other courses, and accordingly, it could not only increase but also expand the sample, i.e., the target population, because a larger scope and number of students would actively use the automatic assessment system.
References
- 1.
UNIZD (University of Zadar), Ocjenjivanje, PowerPoint Presentation. Zadar. 2021. Available from: http://www.unizd.hr/Portals/12/ppt/luka_marinovic/10.%20OCJENJIVANJE.pdf [Accessed: September 08, 2021] - 2.
Wabisabi Learning, The Essential Formative Assessment Tip Guide for Teachers. 2021. Available from: The Essential Formative Assessment Tip Guide for Teachers – Wabisabi Learning [Accessed: September 08, 2021] - 3.
Kyriacou C. Temeljna nastavna umijeća. 2nd ed. Zagreb: Educa; 2001 - 4.
Vizek Vidović V, Vlahović-Štetić V, Rijavec M, Miljković D. Psihologija obrazovanja. Zagreb: IEP-VERN; 2003 - 5.
Stiggins R. Assessment crisis: The absence of assessment FOR Learning. Phi Delta Kappan. 2002; 83 (10):758-765. Available from:https://doi.org/10.1177/003172170208301010 [Accessed: August 10, 2021] - 6.
Earl LM. Assessment as Learning: Using Classroom Assessment to Maximize Student Learning. Thousand Oaks, California: Corwin Press; 2003 - 7.
Watanabe Crockett L, Churches A. Mindful Assessment: The 6 Essential Fluencies of Innovative Learning. Melbourne: Hawker Brownlow Education; 2017 - 8.
Department of Education and Training. 2004. Available from: www.sofweb.vic.edu.au/blueprint/fs1/assess.asp [Accessed: September 08, 2021] - 9.
Holmes-Smith P. Assessment for Learning: Using Statewide Literacy and Numeracy Tests as Diagnostic Tools. Australian Council for Educational Research (ACER); 2005. Available from: https://research.acer.edu.au/cgi/viewcontent.cgi?article=1009&context=research_conference_2005 [Accessed: September 09, 2021] - 10.
Earl L. Rethinking Classroom Assessment with Purpose in Mind; Assessment for Learning, Assessment as Learning, Assessment of Learning. Winnipeg, Manitoba: Manitoba Education, Citizenship and Youth; 2006. Available from: https://digitalcollection.gov.mb.ca/awweb/pdfopener?smd=1&did=12503&md=1 [Accessed: September 09, 2021] - 11.
Fabijanić M, Đambić G, Fulanović B. A novel system for automatic, configurable, and partial assessment of student SQL queries. In: 2020 43rd International Convention on Information, Communication and Electronic Technology (MIPRO 2020). Opatija. 2020. Available from: https://ieeexplore.ieee.org/abstract/document/9245264 [Accessed: August 10, 2021] - 12.
Đambić G, Fabijanić M, Lokas Ćošković A. Automatic, configurable, and partial assessment of student SQL queries with joins and groupings. In: 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO 2021), Opatija. 2021 - 13.
Wagner PJ. The SQL file evaluation (SQLFE) tool: A flexible and extendible system for evaluation of SQL queries. In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 2020. p. 1334. Available from: https://dl.acm.org/doi/10.1145/3328778.3372599 [Accessed: August 10, 2021] - 14.
Al-Salmi A. Semi-automatic assessment of basic SQL statements. [Ph.D. Thesis]. Loughborough University. 2019. Available from: https://repository.lboro.ac.uk/articles/thesis/Semi-automatic_assessment_of_basic_SQL_statements/9730115/1 [Accessed: August 10, 2021] - 15.
Kleerekoper A, Schofield A. SQL tester: An online SQL assessment tool and its impact. In: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education. 2018. p. 87-92. Available from: https://www.researchgate.net/publication/325901267_SQL_tester_an_online_SQL_assessment_tool_and_its_impact [Accessed: August 10, 2021] - 16.
Vichianroj P, Trongratsameethong A. ASQLAG - automated SQL assignment grading system for multiple DBMSs. Journal of Technology and Innovation in Tertiary Education. 2018; 1 (1):41-59. Available from:https://jti.siamtechno.ac.th/index.php/STC-JTI/article/view/14456 [Accessed: August 10, 2021] - 17.
Kleiner C, Tebbe C, Heine F. Automated grading and tutoring of SQL statements to improve student learning. In: Proceedings of the 13th Koli Calling International Conference on Computing Education Research. 2013. p. 161-168. Available from: https://dl.acm.org/doi/10.1145/2526968.2526986 [Accessed: August 10, 2021] - 18.
Brusilovsky P, Sosnovsky S, Lee DH, Yudelson MV, Zadorozhny V, Zhou X. An open integrated exploratorium for database courses. Acm Sigcse Bulletin. 2008; 40 (3):22-26. Available from:https://dl.acm.org/doi/10.1145/1597849.1384280 [Accessed: August 10, 2021] - 19.
De Raadt M, Dekeyser S, Lee TY. Do students SQLify? improving learning outcomes with peer review and enhanced computer-assisted assessment of querying skills. In: Proceedings of the 6th Baltic Sea Conference on Computing Education Research: Koli Calling 2006. 2006. pp. 101-108. Available from: https://dl.acm.org/doi/10.1145/1315803.1315821 [Accessed: August 10, 2021] - 20.
Russell G, Cumming A. Automatic checking of SQL: Computerised grading. In: Proceedings of the Twelfth International Conference on Learning. Granada, Spain: International Journal of Learning; 2005. p. 127-134. Available from: https://www.researchgate.net/publication/237117757_Automatic_Checking_of_SQL_Computerised_Grading [Accessed: August 10, 2021] - 21.
Sadiq S, Orlowska M, Sadiq W, Lin J. SQLator: An online SQL learning workbench. In: Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. 2004. p. 223-227. Available from: https://www.researchgate.net/publication/220808351_SQLator_an_online_SQL_learning_workbench [Accessed: August 10, 2021] - 22.
Coleman Prior J. Online assessment of SQL query formulation skills. In: Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20. Australia: Australian Computer Society, Inc.; 2003. p. 247-256. Available from: https://dl.acm.org/doi/10.5555/858403.858433 [Accessed: August 10, 2021] - 23.
Shieh C-J, Yu L. A study on information technology integrated guided discovery instruction towards students’ Learning achievement and Learning retention. Eurasia Journal of Mathematics, Science & Technology Education. 2016; 12 (4):833-842. Available from:https://doi.org/10.12973/eurasia.2015.1554a [Accessed: August 10, 2021] - 24.
Babić T. Communication paradigm in higher education under the influence of social media. Communication management review. 2021; 06 (01):46-67. DOI: 10.22522/cmr20210164. Available from:https://hrcak.srce.hr/261006 [Accessed: September 09, 2021] - 25.
Grosseck G. To use or not to use web 2.0 in higher education? Procedia Social and Behavioral Sciences. 2009; 1 :478-482. Available from:https://www.sciencedirect.com/science/article/pii/S1877042809000895?via%3Dihub [Accessed: August 10, 2021] - 26.
Mabić M. Društvene mreže u obrazovanju: Što misle studenti Sveučilišta u Mostaru. In: Društvo i tehnologija 2014 – Dr. Juraj Plenković, XXI. Međunarodni znanstveni skup, Opatija. 2014. pp. 74-378. Available from: https://www.researchgate.net/publication/268208102_Drustvene_mreze_u_obrazovanju_Sto_misle_studenti_Sveucilista_u_Mostaru [Accessed: October 29, 2018] - 27.
Raut V, Patil P. Use of social media in education: Positive and negative impact on the students. International Journal on Recent and Innovation Trends in Computing and Communication. 2016; 4 (1):281-285. Available from:https://ijritcc.org/download/conferences/ICRRTET_2016/ICRRTET_Track/1455261816_12-02-2016.pdf [Accessed: August 10, 2021] - 28.
Babić T, Vunarić I, Lokas Ćošković A. Interakcija i komunikacija studenata prilikom učenja – primjer studenata Visokog učilišta Algebra. In: Biljanović P, editor. MIPRO Proceedings. Opatija. 2017. pp. 1165-1169. Available from: https://www.bib.irb.hr/984856 [Accessed: October 10, 2018]