Learning Loss During the COVID-19 Pandemic and the Impact of Emergency Remote Instruction on First Grade Students’ Writing: A Natural Experiment By: Gustaf Bernhard Uno Skar, Steve Graham and Alan Huebner

The coronavirus disease 2019 (COVID-19) pandemic and the sudden cancellation of in-class instruction for many students around the world presented an unprecedented disruption in children’s education. As the COVID-19 pandemic took form, multiple concerns were raised about the potential negative impact on students’ learning. The current study examined this proposition for children’s writing. We compared the quality of writing, handwriting fluency, and attitude toward writing of first grade Norwegian students during the COVID-19 pandemic (421 girls, 396 boys), which included emergency remote instruction for almost 7 weeks, with first grade students in the same schools a year before the pandemic began (835 girls, 801 boys). After controlling for variance due to national test scores, school size, proportion of certified teachers, students per special education teacher, school hours per student, student gender, and native language, we found that students attending first grade during the pandemic had lower scores for writing quality, handwriting fluency, and attitude toward writing than their first grade peers tested a year earlier before the COVID-19 pandemic emerged. Implications for policy and instruction as well as future research are presented. 

Educational Impact and Implications Statement
 In December 2019, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus was identified, and it spread across the world quickly causing the coronavirus disease 2019 (COVID-19) pandemic, presenting unprecedented challenges for the education of school-age children. This study examined the impact of the pandemic and the temporary 7 week replacement of in class learning with online instruction on the writing of first grade children in Norway. First grade students tested shortly after the end of online instruction displayed a learning loss, having lower scores on measures of writing quality, handwriting fluency, and attitude toward writing than first grade children from the same schools tested a year earlier before the start of the pandemic.

 In December 2019, a new virus, SARS-CoV-2, was identified. It spread across the world quickly, presenting unprecedented economic and social challenges, impacting virtually all aspects of daily life including public health, the world of work, and food systems.(1) To help slow the spread and impact of the virus, countries across the globe in March and April of 2020 moved from in class instruction to emergency remote instruction, which included remotely delivered instruction (e.g., online, radio) or some combination of remote and in class instruction (Di Pietro et al., 2020; Hodges et al., 2020). 

Norway, the country in which this study took place, was among the nations that canceled in class instruction and applied a model of emergency remote instruction that was delivered online in March of 2020. In a public statement on March 12, the Norwegian Directorate of Health announced that it had decided to close all schools immediately. The closing of schools, which was replaced by emergency remote instruction via digital meeting platforms, was to last until March 26, 2020. (2) Subsequently, the Norwegian government extended the period of emergency remote instruction to April 27. The lockdown lasted for 46 days, weekends and Easter holidays
 included. (3)  This sudden cancellation of in-person instruction in Norway and across the globe presented an extraordinary educational challenge. It was estimated that 1.19 billion of students across the world were impacted (UNESCO, 2020).

The world had never faced an educational disruption of this magnitude (Azoulay, 2020; Winthrop, 2020). The application of emergency remote instruction, including the use of online platforms to promote such instruction, provided a mechanism for ensuring the continuity of schooling around the world (Di Pietro et al., 2020). Even so, many experts expressed concerns about the effects of this sudden transition on students’ learning (e.g., Daniel, 2020). While data on the actual learning loss experienced by students is sorely needed (the focal point of this investigation), predictions about the level of learning loss and its subsequent consequences proliferated.

Citing forthcoming research by Hanushek and Wosseman, Schleicher (2020) noted that a loss of one third of a year of schooling due to the COVID-19 pandemic would represent a 1.5% loss of current gross domestic product (GDP) for a typical country for the rest of this century. For the United States, this would translate to a total loss of $15,300,000,000,000 by the year 2100. Consequently, it is critical to acquire information on students’ learning loss as a result of the COVID-19 pandemic and the resulting emergency remote instruction. There is little reason to expect all aspects of student learning were impacted equally, as learning effects differ from one academic domain to the next (Alexander, 2003). As a result, this study focused on one aspect of students’ learning: writing.

 Why is it important to examine the association between learning to write and the COVID-19 pandemic and the ensuing emergency remote instruction? Writing is a foundational skill in terms of students’ success at school and beyond. Writing is one of the primary means by which students’ demonstrate what they know (e.g., written essays, written responses on tests). Writing about material read enhances comprehension of the ideas presented in text, and writing instruction makes students better readers (Graham & Hebert, 2011). Writing also improves learning, as writing about content material makes material more understandable (Graham et al., 2018). Just as important, writing is a versatile tool as we use it to record information, communicate, persuade, inform, chronicle experiences, think about topics, create imaginary worlds, entertain others, explore the meaning of events, and heal psychological and physiological wounds (Graham, 2006).

Why the COVID-19 Pandemic and Emergency Remote Instruction Were Likely to Reduce Student Learning

Stress

 There are multiple reasons why the COVID-19 pandemic and emergency remote instruction might negatively impact learning, including learning to write. One possible determinant to students’ learning involves the effects of the COVID-19 pandemic and ensuing emergency remote instruction on teachers and parents. Teachers and parents have reported increased stress as a result of schools moving to distance learning (Bonal & Gonzales, 2020; Kimm & Ashbury, 2020). As teacher stress increases, there is a corresponding decrease in student learning (e.g., Oberle & Schonert- Reichl, 2016). Additionally, parents have become proxy educators during emergency remote instruction—a responsibility they cannot easily abdicate.

Davis et al. (2021) reported that stress is especially elevated for parents of children who find distance learning challenging, as occurred in many countries such as Norway during emergency remote instruction. This increased stress is likely to impact their children’s learning. The potentially negative effects of stress on learning are not limited to teachers and parents, but extend to children too. A study by Sprang and Silman (2013) found that children who were isolated at home during previous pandemics were more likely to experience stress disorder, grief, and adjustment problems. Howard and Howard (2012) also reported children experienced increased anxiety and fear during a prior pandemic.
Higher levels of stress, anxiety, and fear take a toll on students’ well-being and learning, and can result in poorer learning outcomes (Di Pietro et al., 2020).

Resources for Distance Learning

Emergency remote learning that accompanied the COVID-19 pandemic required that teachers deliver instruction using distance learning methods. In some instances, this involved older technology such as radios (Kirshner, 2020), and in other instances it involved newer digital learning platforms (Fauzi & Khusuma, 2020). While digital tools can enable and support learning and writing in multiple ways (Graham, in press a), the COVID-19 pandemic and ensuing in-person school closures occurred at a time when many countries, schools, and teachers were not ready to apply these tools effectively.

For example, just before the pandemic, one-fourth of principals in countries participating in the Organization for Economic and Cooperation Development (OECD) assessments indicated learning was hindered due to a shortage or inadequacy in digital tools (Schleicher, 2020). Many teachers in this OECD survey also indicated the need for additional training in the use of digital tools, and 47% of teachers did not allow students to use these tools for classwork. For these teachers, the quick transition to emergency remote instruction using digital tools that occurred as a result of the COVID-19 pandemic raises questions about their effectiveness in delivering instruction digitally and remotely.

The use of digital platforms for student learning at home during the COVID-19 pandemic not only depends on teachers’ skills and school resources; they rely on parents’ availability and abilities to serve as proxy educators as well as the availability of digital resources at home. Such resources are not equally available to all families. Students from less affluent homes are less likely to have access to a laptop or computer, family members are less likely to be able to telework from home, and their parents may be not be as educated (Di Pietro et al., 2020). These factors likely diminish the effectiveness of digital learning for some students.

Quality of Learning Opportunities

 One potentially negative consequence of moving education to a remote learning environment as was done with emergency remote instruction during the COVID-19 pandemic is that less time may be devoted to learning (Schleicher, 2020). For instance, Huber et al. (2020) reported that weekly learning time was 4 to 8 hr less during emergency remote instruction during the COVID-19 pandemic in Austria, Switzerland, and Germany than when students attended schools in person. Theoretically, learning is a consequence of time spent and time needed to learn (Carroll, 1989). Lowering instructional time for writing (Graham & Santangelo, 2014) or time spent writing (Graham et al., 2015) is likely to have a negative impact on writing.

 In addition to instructional time, other factors can potentially decrease students’ learning during emergency remote instruction. For example, while students can interact with others when learning digitally, this may not be as effective as the contact students experience during in class instruction (Di Pietro et al., 2020). Likewise, students may be less motivated to learn due to decreased instructional time and less direct person-to-person contact with their teachers. Finally, some complex skills like writing may be easier to learn when teachers have direct access to students.

Purpose of the Present Study

 This study examined the effect of the COVID-19 pandemic and emergency remote instruction on the writing of first grade students in Norway. Because it is not ethically responsible to assign students to in class and remote learning during a pandemic, the impact of such a situation must be studied with more natural methods. Consequently, we assessed the quality of first grade students’ writing, handwriting fluency, and attitude toward writing in May/June of 2019 before the first case of the COVID-19 pandemic was reported (referred to as the before COVID-19 cohort). In May/June of 2020, we tested a second cohort of first grade students from the same schools using the same writing measures (referred to as the during COVID-19 cohort).

This was after the start of the pandemic and just following the end of emergency remote instruction in Norway (students returned to in-class instruction on April 28, 2020). This allowed us to assess the writing of comparable cohorts of first grade students who did and did not experience these the COVID-19 pandemic conditions. This is the first study to our knowledge that provides an estimate of the effects of the COVID-19 pandemic and emergency remote instruction on children’s writing. We focused our attention on first grade as there is a general consensus that problems that occur early in a child’s education, if not corrected quickly, can become more intractable over time (Slavin et al., 1989).

While a previous survey study by Håland et al. (2019) indicated that close to one in five teachers do not teach writing in the first semester of first grade, a larger study by Graham et al. (2021) just before the COVID-19 pandemic began found that virtually all first grade teachers taught writing. Teachers in this second study indicated they used a variety of instructional procedures to teach writing, and their students spent about 20 min a day writing.

Thus, first grade provided an excellent point to study the effects of the COVID-19 pandemic and emergency remote instruction. This is the initial point where writing is taught to virtually all students in Norway, and negative effects on writing due to the COVID-19 pandemic and emergency remote instruction with first graders are likely to reverberate across the school years. The theoretical framework that guided the current study was the Writer(s)-within-Community model (WWC; Graham, 2018a, 2018b, in press b).

According to this model, writing and writing instruction are simultaneously and interactively shaped and bound by the writing communities in which they occur as well as the cognitive capabilities and resources of the members of these communities (teachers and students). In terms of students’ capabilities and resources, the model posits that students’ motivations for writing, including attitude toward writing, fuel effort and provide the impetus for students to draw upon available cognitive control mechanisms (executive functioning processes) to regulate and process information for writing from long-term memory and external sources as well as execute the processes needed to produce writing (conceptualization, ideation, translation, transcription, and reconceptualization). 

For writers in general, and beginning writers particularly, the cognitive resources specified in the WWC model are limited (Graham, 2018a, 2018b). This is especially problematic for first graders because their use of writing production processes requires conscious attention and considerable cognitive resources.

As a result, young children commonly minimize conceptualization (e.g., planning) and reconceptualization process (e.g., revising) as they write, applying an approach that involves generating one idea that serves as the stimulus for the next idea, and so forth. They apply this approach, at least in part, because transcription skills (e.g., handwriting) are so slow and effort intensive that engaging in other resource intensive activities
 such as conceptualization and reconceptualization can result in cognitive overload.

 In the current study, we included a measure of attitude toward writing as well as handwriting fluency. As noted above, the former can impact young writers’ effort and persistence (Graham, 2006), whereas the latter impacts written production (Berninger, 1999). We also assessed the quality of students’ writing on a functional and authentic writing task. The WWC model (Graham, in press b), along with practical constraints, influenced our decision to focus on a single country in this study. The model posits that writing and writing instruction are shaped and bound by local context (writing purposes, available writing tools, typical writing practices, types of written products, experiences and resources of classroom members, physical and social environment, and collective history of the classroom) as well as macro-factors including cultural, social, institutional, political, and historical influences.

While the model supports the proposition that no single writing classroom or its effects are exactly identical within a country, it also stands to reason that classrooms that share a similar set of cultural, social, institutional, political, and historical catalysts are more likely to have similar effects on writing than ones that operate under a different configuration of mitigating influences. We made the decision to examine the effects of the COVID-19 pandemic and emergency remote instruction in Norway because it is a country with a strong educational system (scores above average on OECD assessments) and digital infrastructure (98% of homes have Internet access), where the use of digital technology for teaching has been emphasized for decades (Blikstad-Balas et al., in press).

It is also an affluent country, identified as the fourth richest nation in the world (https://data.oecd.org/gdp/gross-domestic-product-gdp.htm). This provided a strong test of the hypothesis of the negative effects of the COVID-19 pandemic and emergency remote instruction, as Norway had the infrastructure and resources to respond strongly to this crises.

Research Question and Prediction

Our study answered the following question: Did the COVID-19 pandemic and emergency remote instruction negatively impact the quality of first grade students’ writing, handwriting fluency, and attitude toward writing? We predicted that the cohort of students who attended first grade during the COVID-19 pandemic and emergency remote instruction would evidence lower scores for writing quality, handwriting fluency, and attitude toward writing than first grade cohort of students attending the same school a year earlier before the COVID-19 pandemic.

Because no two sets of students are exactly the same from one year to the next, our analysis controlled for variance due to school performance on national tests (these are provided to older students in Norwegian schools), school size, proportion of certified teachers, number of students per special education teacher, average school hours spent teaching students, student gender, and primary languages of students (native Norwegian speaker, bilingual speaker with Norwegian as a native language, and native speaker of a language other than Norwegian).

In the context of this study, bilingual refers to students who learn two or more languages as part of their normal development. For example, bilingual student would be one who learns Norwegian and Swedish as they are growing up, as both
 languages are spoken naturally at home.

School variables like the ones included in this study (Walberg & Ethington, 1991) as well as student gender and language predict students’ writing performance (Camping et al., 2020; Cordeiro et al., 2018). In terms of language, we expected that students who were native Norwegian speakers and bilingual speakers with Norwegian as a native language would likely be better writers than native speakers of a language other than Norwegian because students were asked to write in Norwegian (see Camping et al., 2020, for evidence to support this contention).

Additionally, we expected that girls’ scores on all three writing measures would be higher than those of boys. Gender is associated with individual differences in writing quality, handwriting fluency, and attitude toward writing, as girls produce better text, have faster handwriting, and express more positive attitudes about writing (e.g., Ekholm et al., 2018; Graham et al., 1998; Reilly et al., 2019). Our prediction that the COVID-19 pandemic and emergency remote instruction would depress students’ writing scores was based on three arguments. One, as noted earlier, quality of learning, resources for distance learning, and stress created by the COVID-19 pandemic and the resulting emergency remote instruction provided are likely to dampen students’ learning.

 Two, Blikstad-Balas et al. (in press) investigated how a large sample of parents (N = 4,642) viewed the success of emergency remote instruction in Norway. While this study included parents of children in Grades 1 to 10, we only share findings for Grades 1 to
 4 here because they are most relevant to the current investigation. Twenty-three percent of parents in Blikstad-Balas et al. (in press) reported that their children were not expected by teachers or schools to attend classes online at all.

Another 27% of parent indicated their children demonstrated attendance by completing assigned tasks. According to parents, 7%, 26%, and 21% of children had no teacher contact, contact once a week, or contact two or three times a week during emergency remote instruction. Many parents (51%) strongly agreed they had to continuously monitor their child’s progress to ensure schoolwork was completed, and 76% of them indicted they had to spend more time than usual following-up on their child’s schoolwork (85% of parents reported spending 1 to 2 hr more on their child’s schoolwork a day).

Even more important, 23% and 35% of parents indicated their child spent 50% or less than 50% of time, respectively, doing schoolwork during emergency remote instruction. Therefore, many first graders had little contact with their teachers during emergency remote instruction, and the time devoted to learning was severely restricted for many of them. This makes it likely that students’ writing development during the pandemic and resulting emergency remote instruction would be depressed. This is especially important for the writing measures included in our study, as handwriting fluency develops mainly through practice writing (Graham, 2009), quality of writing is impacted by how much students’ write and the instruction they receive (Graham et al., 2012),
 and attitude toward writing is influenced by interactions with teachers and peers (Graham et al., 2015). 

Finally, most studies examining possible decrements in writing when school is not in session for summer break have reported loss in writing skills (Jesson et al., 2009; Pacchter et al., 2015). This occurred even when the break was only 6 weeks long (Meyer et al., 2020). While the conditions between a summer break and the COVID-19 pandemic and emergency remote instruction are not identical, these findings provided support for our prediction. To better situate the possible learning loss in writing experienced by grade one students in the during COVID-19 cohort, we draw on the findings from an earlier study where we collected data on the same three writing measures (writing quality, handwriting fluency, and attitude toward writing) with second-grade students in the same schools (Graham et al., 2021).

This data was collected at the exact same time as data for the before COVID-19 sample in this investigation (May/June 2019). These data provided a benchmark for what students in the during COVID-19 cohort should have achieved, and this is explored in the Discussion.

Method

Participants

This study included 2,453 first grade students in Norway (also see Table 1). Two-thirds of these students (N = 1,636) attended first grade in the 2018–2019 school year (before COVID-19 cohort). This was the school year before the COVID-19 pandemic, and all students were taught at school for the full school year. The remaining participants (N = 817) attended first grade during the year of the COVID-19 pandemic in 2019–2020 (during COVID-19 cohort). These students were not taught at school from March 12 to April 27 due to the institution nationwide of emergency remote instruction in Norway. These two cohorts (before and during COVID-19) represented 4% of all first grade students in Norway across the 2 years combined.

 The participating students came from 200 classrooms, in 60 schools, in four municipalities. Two of the municipalities included major Norwegian cities, while the other two were more rural. These schools represented 2% of all schools in Norway. The average school size for the before and during COVID-19 cohorts were 462.1 (SD = 170) and 439.3 (SD = 164.2), respectively. The average class in the before COVID-19 cohort received 55 teacher hours (SD = 11.8) per student, whereas the average class in the during COVID-19 cohort received 55.4 teacher hours (SD = 8.6).

This metric is the number of instructional hours divided by the number of students. Most of the first grade teachers were certified: 96.4% in the before COVID-19 cohort (SD = 5.1) and 95.5% in the during COVID-19 cohort (SD = 5.1). On average, there were 88.2 students (SD=35.23) per special education assistant in the before COVID-19 cohort in each school and 92.9 (SD= 34.1) in the during COVID-19 cohort. The average score on national tests for schools for the before COVID-19 cohort students was 51.2 (SD=2.9); it
 was 50.6 (SD=2.3) for the during COVID-19 cohort. Refer to Table 2 for a summary of these statistics.

 Multiple indices provided evidence that the participating students in the before and during COVID-19 cohorts were representative of students nationally. While there are no national, standardized tests in writing in Norway, tests of reading, mathematics, and English are administered in fifth, eighth, and ninth grade for students in compulsory school (i.e., Grades 1–10).

The participating school's average scores across the two cohorts for the three subjects was 50.9 (SD= 2.7), where averages for the before and during COVID-19 cohorts were 51.2 (SD=2.9) and 50.6 (SD=2.3), respectively. The population average is 50.0, with a standard deviation of 10. Furthermore, the proportion of boys and girls in the before and during COVID-19 cohorts were similar to the proportion in the Norwegian population.

In 2018–2019, 48.6%of the population of Norway were girls, and 51% of the before COVID-19 cohort were girls (within the 95% confidence interval [CI]of the sample estimate of this study). In 2019–2020, 48.9% of the national population were girls, and 51.5% of the during COVID-19 cohort students were girls (within the 95% CI of the sample). Although there are no publicly available registers of language backgrounds in Norway, 8% of students in 2018–2019 were entitled to extracurricular language instruction, and 7.9% of students received such instruction in 2019–2020. These statistics are relatively close to the 5.5% and 6.7% of students in the before and during COVID-19 cohorts, respectively, who did not speak Norwegian as their first language. 

While the size of schools from which students in the before and during COVID-19 cohorts combined were drawn (M= 454; SD=168.4), was larger than the national average size of schools (M=225,SD=166), the four municipalities participating in this study were representative of the population range of municipalities in Norway. It ranged from large municipalities (population of 673,469, representing 13% of the population) to average sized municipalities (population of 13,958, representing 3% of the population) to smaller municipalities (population of 6,882, representing 1% of the population). Additionally, the proportion of certified teachers (96% average across both cohorts) and the school hours per student (55 hr across both cohorts) for the teachers teaching the first grade students in this study were similar to national average, which was 95.% and 61 hr, respectively. Also, the average number of students per special education teacher (88.2 across the before and during COVID-19 cohorts) was similar to the national average, which was 83.8 (SD = 98.4).

Sampling Procedures

The recruitment of students was done in the context of a large-scale writing intervention study (Skar, Aasen, & Jølle, 2020). First grade students in the before COVID-19 cohort participated in a single test administration in May/June of 2019. This occurred before the COVID-19 pandemic and before schools were assigned to treatment and control conditions. All of the first grade students tested at this time were included in the before COVID-19 cohort. 

None of these students participated in a writing intervention before they were tested. Students in the during COVID-19 cohort were assessed on the same writing measures in May/June of 2020. Due to COVID-19, schools in Norway moved from in class to digital instruction about 5 weeks before this group of students was tested. The during COVID-19 cohort only included first grade students from the 22 schools assigned to the control condition. Including students from schools receiving the writing treatment would have biased any comparison involving first grade students from the before and during COVID-19 cohorts.

Because the during COVID-19 cohort only included control students, it included about one half of the students included in the before COVID-19 cohort. The sampling scheme was motivated by the desire to obtain the largest sample size possible, as statistical power is always a consideration. Moreover, as we explain below, multilevel regression modeling was the main statistical methodology used for this study, and we wished to ensure there were no issues with the stability of the model estimates.

Thus, measurements from all students were included in the before COVID-19 cohort, as students had not yet been assigned to or received treatment (i.e., writing intervention program). Moreover, there were no statistically significant differences for writing quality, handwriting fluency, and attitude toward writing, after controlling for clustering, between students who were eventually assigned to the writing intervention and control group in the before COVID-19 cohort (all ps . .31).

However, we only included students from the control condition in constructing the during COVID-19 cohort. Like students in the before COVID-19 cohort, children in the during COVID-19 cohort were taught writing as usual. This made these
 children as similar as possible to the children in the before COVID-19 cohort, as they attended the same schools and received a business as usual approach to writing, with the exception that they were enrolled in school during the COVID-19 pandemic and emergency remote instruction.

This difference was the focal point of this investigation. The schools participating in this and the larger study (Skar, Aasen, & Jølle, 2020) were recruited with the help of executive officers from different municipalities. All schools in a municipality were invited to participate. Students of participating schools were invited to participate by means of parental consent. In the before COVID-19 cohort, 2,459 students were approached, and 2,294 (93.4%) consented to participation. In the during COVID-19 cohort, 1,333 students were approached, and 1,170 (87.8%) consented to participation. For the before and during COVID-19 cohorts, respectively, 71% and 69.8% of participating students participated in all test administration, resulting in the net sample of 1,636 and 817 for the two cohorts.

Writing Measures and Covariates

 Students in both the before and during COVID-19 cohorts completed the same three writing tasks. This included writing a letter that yielded a measure of writing quality, a copying task which provided an assessment of handwriting fluency, and a short survey
that measured attitude toward writing. There were a total of eight covariates included in the analysis (described below.)

Writing Quality 

Students wrote a letter using a prompt developed by the Functional Writing in Primary School project (Skar, Aasen, & Jølle, 2020). Teachers introduced this writing task by reading aloud a letter from researchers at the local university. The letter explained that these researchers were interested in children’s favorite recess activities. It further indicated students could help researchers at the University to understand this topic better by writing “a letter where you describe what you usually play with during recess
 time.” Before writing the letter, teachers led a discussion with students about what researchers do, what students like to do during recess, why they like each activity, what it means to describe something, and what is included in a letter. Teachers also showed students a picture of a school playground with children engaged in multiple activities (e.g., kicking a ball, playing marbles, swinging, or building a sandcastle). Students were informed they had 45 min to write their letter. We asked teachers not to provide students with help as they wrote.

Students’ letters were scored for writing quality on the following eight traits: audience awareness, organization, content relevance, vocabulary, sentence construction, spelling, legibility, and punctuation. Each trait was scored using a separate 1 to 5 rating scale. The rating scaled were developed and validated for use in Grades 1–3bySkar, Jølle, and Aasen (2020), and also used in Skar et al. (2021), who reported that, on average, girls outperformed boys, and students in higher grades outperformed students in lower grades.

Higher scores represented better quality for the trait assessed. The scores for the eight traits were summed into a single, scaled score. The rating scale for each writing attribute had descriptors and benchmark text for each score point (1 to 5). The rating scale “audience awareness” focused on how text explicitly or implicitly addressed a reader. The descriptor for a score of 5 (the highest score) stated that: “The text addresses the reader in the assignment in a generally relevant manner and takes into account the reader’s need for knowledge of participants/characters, circumstances, and
 events.

The text may contain traces of the student’s voice with reflective or evaluating utterances.” For a score of 1 (the lowest score), the descriptor stated that “To understand the text, a conversation with the writer is required.” The 1–5 point scale for “organization” concerned the internal structure of the text at the macro-, meso-, and microlevel. “Content relevance” focused on whether the letter presented relevant content with regards to the writing task assigned; a higher score indicated more relevant content, but not necessarily longer text. “Vocabulary” involved the repertoire
 of words used in the text, with the highest level descriptor stating that: “The text contains a repertoire of words and expressions that are relevant to the task. In some cases, there is use of specialized and abstract words, and/or creative forms of expression.”

“Sentence construction” concerned syntactical variation and syntactic complexity, whereas “spelling” concentrated on increasing mastery of correct spelling in text. “Legibility” focused on the legibility and shape of letters, including if letters were reversed or not. Finally, “punctuation” involved the correct used of punctuation marks. The descriptor for the lowest level stated that: “the text does not contain punctuation marks,” whereas the descriptor for the highest level stated that: “mostly correct use of different punctuation marks. The use of comma may occur.”

Descriptors for scores of 1 and 5 for all scales are presented in the Appendix. The quality of students’ letters were scored by a pool of 34 trained raters at two time points. Papers produced by students in the before COVID-19 cohort were scored after these students
 were tested as were papers generated by students in the second cohort (i.e., during the COVID-19 pandemic). These raters were recruited from staff and master-level students in teacher education at the University where the first author was employed. In the training sessions, the rating scales were presented and practiced, and each rater was given opportunities to address any questions that arose while practicing. Further, the raters sent their first 25 ratings to the first author who provided feedback on individual
 raters’ relative harshness and consistency.

There was also an email list to which the raters could address questions that arose during the rating session. When a question was posted on the list, ratings stopped, and raters were given directions addressing the question posed. Each text was assessed independently by two raters. To provide an empirical link between raters and rating occasion, fifty of the students’ letters were scored by all raters. This empirical link was used to scale the raw scores from the two occasions with a many facet Rasch model (MFRM; Linacre, 2017). The following model was used:

 where Pnij(k) represents the probability of student n, rated on rating scale i, by rater j, receiving a score of k, and Pnij(k-1) represents the probability of the same student under the same conditions receiving a score of (k-1). Bn is the ability for person n, Ei is the difficulty of rating scale i, and Cj is the severity of rater j. Lastly, Fx represents the point on the logit scale where category k and (k-1) are equally probable. (4)

The data from the two occasions fit the model well. The “reliability of separation” measure, which is a MFRM analog to Cronbach’s a was .94 for papers from the before COVID-19 cohort and .96 for papers from the during COVID-19 cohort, which indicate a
 reliability fit for high stakes assessment in the measure of students’ writing quality. The standardized residuals also indicated good fit, as 3.90% were in the range of 2–3 and .59% exceeding 3 for papers from the before COVID-19 cohort, and 4.10% were in the
 range of 2–3 and .80% exceeding 3 for papers from the during COVID-19 cohort.

 Standardized residuals above 2 should not exceed 5%, and standardized residuals above 3 should not exceed 1% (Eckes, 2011). We used the MFRM estimates to adjust scores using a compensatory principle where the final writing quality score was a single averaged score adjusted for varying difficulties between rating scales and differing rater severity. The FACETS software (Linacre, 2017) was used to convert the MFRM estimates
 to a Text Quality Scale, where scores were not limited to integers.

Handwriting Fluency

Teachers administered a copying task designed to assess students’ handwriting fluency (i.e., rate at which letters of the alphabet were produced correctly). Students were directed to copy a short paragraph as quickly and accurately as possible within a 90 s period (Graham et al., 1997). The paragraph was from the group Diagnostic Reading and Aptitude and Achievement Tests (Monroe & Sherman, 1966). Students were shown an instructional video that explained to students how the copying task was to be completed. After watching the video, the students followed along as the teacher read the paragraph aloud. With the written paragraph in front of them, students were told to copy the paragraph once the teacher said “Start,” and to stop copying when directed
 to do so at the end of 90 s. While the paragraph students were asked to copy was written in article, students were not asked to copy it in article or another script such as cursive. There is no specific mandate in Norway to teach article or cursive handwriting. This is determined by teachers.

 The score for handwriting fluency was obtained by tallying all letters from the paragraph written correctly in the 90 s interval. The trained coders did not count as correct letters incorrectly written, skipped, or correctly written but not a match to text. To estimate interrater reliability, 10% of all tasks were rescored by an independent coder. Reliability was satisfactory (j = .812, intraclass correlation coefficient [ICC] = .99). The score for handwriting was divided by 1.5 to provide an estimate of handwriting
 fluency per min.

Attitude Toward Writing

 Students completed a short survey that included four items designed to measure attitude toward writing. The items were: “I liked the composition task,” “I am satisfied with my text,” “I am satisfied with my effort,” and “I like to write.” Students rated each statement using a 3-point Likert-type scale. The highest point on the scale was denoted by three stars (higher attitude toward writing), while the lowest point was a single star (lower attitude toward writing). An exploratory factor analysis, using principal axis factoring, resulted in a one-factor solution, with all items loading at .58 or higher. This factor accounted for 53.80% of the variance (coefficient a = .71). The score for the attitude toward writing measure was the average score of all four items.

Covariates

 The following covariates at the school-level were included in the analysis: national test result, school size, proportion of certified teachers, students per special education teacher, and school hours per student. Data for these variables, which were described earlier were collected from the database “Skoleporten” [The School Gate], hosted by the governmental agency the Norwegian Directorate for Education and Training (https://skoleporten.udir.no/). Covariates at the student-level were gender and language background. At student level, we also included a student cohort membership variable (before COVID-19 cohort and during COVID-19 cohort). (5) Data on gender and language background were collected by participating students’ teachers who identified each student’s gender and whether the student learned Norwegian or another language first, or both.

A student was coded as having learned Norwegian first (denoted “L1”), another language than Norwegian first (denoted “L2”), or Norwegian and another language from birth (denoted “bilingual”). The teacher did not assess the students’ language skills when noting language background, which means that the level of proficiency within the L2- and bilingual groups might be quite varied. Table 3 presents the descriptive statistics for student level measures. We were unable to collect data on school-level variables for 36 students. To refrain from excluding them from the analysis a mean imputation strategy was used. We believe this strategy was reasonable given the relatively small proportion of missing data.

Procedures

 Procedures to collect data were designed to fit the Norwegian educational context. A  testing and assessment is controversial in Norway and formal grades are not introduced until Grade 8, we developed test procedures that would mimic regular instruction as much as possible, while allowing the collection of data under uniform conditions. Other reasons why we mimicked regular practices is that formal instruction in first grade has been a topic of public and heated debate in Norway (e.g., Lundetræ, 2018), and the transition from kindergarten to school was a topic of a relatively recent report from the Norwegian Knowledge Center for Education (Lillefjordetal.,2015).

Thus, we did not want to upset teachers, students or guardians by introducing unusual formats in the classrooms. The first author consulted with first grade teachers to develop a strategy for administering the three writing tasks that would balance the needs of contextual alignment and standardization. It was decided that writing tasks would be administered during a regular lesson, and students would be offered 45 min to complete the letter writing tasks, which is similar to the timeframe students apply to tasks that are part of regular class instruction in Norway.

The copying task was also administered as part of regular instruction, but only took 1.5 min to complete it. We provided teachers with instructions for administrating the three writing tasks. This included written and video instruction for each task. The video instruction for the copying task was shown to students. Task administration was counter balanced so that one half of the teachers administered the letter writing task and attitude toward writing measure first and the copying task second, whereas the remaining teachers did just the opposite. After administering the tasks to students, teachers posted test materials on a secure site accessible only by the researchers. The data was collected on two occasions (May/June2019andMay/June2020). On both occasions, teachers had a 15-day window to complete data collection.

Results

Power Analysis

 The analytical strategy applied in this investigation was multilevel, or hierarchical, regression analysis. For regression models, the estimate of statistical power may focus on inference for a coefficient, a variance component, or on the means of particular groups (Snijders, 2005). Often times, then focus is on aregression coefficient, and in the current study we focused on the coefficient indicating the effect of the COVID-19 pandemic and emergency remote instruction on students’ scores on each of the writing measures. In multilevel modeling, the statistical power of a test for a given regression coefficient depends on sample size at each level (total number of students, classes, and schools; Snijders & Bosker. 1999) as well as myriad other factors.

Given the computational complexities introduced by multiple levels of nesting, statistical
simulation is a current method of assessing statistical power. Investigations using the MLPowSim software (Browne & Golalizadeh, 2021) to assess statistical power of the regression coefficient for detecting differences between the before COVID-19 and
 during COVID-19 cohorts indicated high power (.99%) for the current sample sizes. The simulation was based on estimates derived from results from a study by Skar et al. (2021), who reported on similar measures.

Analytical Strategy

 For each writing measure (writing quality, handwriting fluency, and attitude toward writing; Table 3), a three-level linear hierarchical regression model was computed in which students, classrooms, and schools represented Levels 1, 2, and 3, respectively. The models allowed for random intercepts for classes and schools. For each writing measure, the following three-level hierarchical models were fit: (1) a “null” model (no predictors) to assess the correlation structure via the ICC, (2) a model with all covariates
 except cohort membership (before COVID-19 cohort vs. during COVID-19 cohort), and (3) a model with all covariates including the cohort membership variable.

The purpose of fitting models (2) and (3) was to calculate the difference in explained variation (R2) between the two models, which was interpreted as the increase in R2 Provided 3 Specifically 3 Differences in the writing performance of students in the before COVID-19 cohort and the during COVID-19 cohort. For purposes of interpretation and computational stability, standardized versions of continuous covariates were used in the three models.

Was the COVID-19 Pandemic and Emergency Remote Instruction Associated With Lower Writing Scores?

Table 4 displays effect sizes for all three models fit for each of the three writing outcome variables (writing quality, handwriting fluency, and attitude toward writing). For each outcome variable, Table 4 displays ICCs resulting from the null model (for three level hierarchical models, there are two ICCs: one for Level 2 and one for Level 3) and the R2 values from the models without and with the student cohort membership variable (before COVID-19 cohort vs. during COVID-19 cohort).

It can be seen that writing quality had the highest correlation within classes and schools, while attitude toward writing had the lowest correlation. Also, of the three outcomes, the student cohort membership variable (before COVID-19 cohort vs. during COVID-19 cohort) explained the most variation for writing quality (contributed R2 almost 5%) and the least variation for attitude toward writing (contributed R2 less than 1%). Effect size for R2, f2 = R2/(1 R2), is reported in parentheses next to the R2 values in Table 4. A rule of thumb interpretations for f2 is that values of .02, .15, and .35 represent small, medium, and large effects (Lorah, 2018). Thus, the student cohort variable (before COVID-19 cohort vs. during COVID-19 cohort) had a small to medium effect for writing and quality and handwriting fluency and a small effect for attitude toward writing.