Reliability and validity of a brief assessment tool measuring children’s energetic play in early childhood education and care

Reliability and validity of a brief assessment tool measuring children’s energetic play in early childhood education and care | Research Square Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Reliability and validity of a brief assessment tool measuring children’s energetic play in early childhood education and care Emma K Adams, Hayley Christian, Stewart G Trost This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8859234/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background: Measuring young children’s movement behaviours while attending early childhood education and care (ECEC) services is important for evaluating the effectiveness of physical activity interventions and adherence to ECEC national quality standards. This study evaluated the inter-rater reliability and concurrent validity of a short educator-reported energetic play assessment tool. Methods: Educators from 10 ECEC services reported on energetic play levels across the childcare day for 80 children. Inter-rater reliability was examined using Kappa statistics and intra-class correlation coefficients (ICCs). Concurrent validity was assessed against device-measured physical activity data using partial Peason correlations. Results: Inter-rater reliability for the four time periods across the childcare day ranged from fair for the morning tea to lunch time period (0.39) to substantial for the lunch to afternoon tea period (0.71). The ICC for combined time periods was good at 0.75 (95% CI 0.64, 0.83). Energetic play ratings were not correlated with device-measured energetic play but were positively correlated with total physical activity (rating 1: r =0.24, p =0.046; rating 2: r =0.30, p =0.012). Conclusion: The educator-completed brief energetic play assessment tool had moderate evidence of inter-rater reliability, however, modest correlations with device-based physical activity were observed for total physical activity but not energetic play. Refinements to the tool could improve validity and provide important feedback to inform strategies to meet and exceed ECEC national quality standards around promoting and supporting children’s physical activity. physical activity measurement childcare educator preschool Introduction Regular physical activity is vital for young children’s physical and mental health and development. 1 International guidelines recommend children aged three to five achieve 180 minutes of total physical activity each day, including 60 minutes of energetic play (moderate-vigorous intensity physical activity). 2 However, large proportions of children do not meet these recommendations. 3-6 Many young children spend a considerable amount of time each week in early childhood education and care (ECEC), making this a key behaviour setting outside of the home to influence physical activity levels. 7 In Australia, over 40% of children under five years of age are in childcare and attend for an average of 25 hours per week. 8 Long daycare services in Australia provide care and education for children before entering compulsory schooling. They are regulated through the Australian Children’s Education & Care Quality Authority (ACECQA) National Quality Framework, which includes assessing and rating ECEC services on seven quality areas, including children’s health and safety (Quality Area 2.1: Each child’s health and physical activity is supported and promoted). 9 Overall, there is little physical activity-related resources or training for ECEC services and few services incorporate physical activity in their service policies. 5, 10 In addition, recent evidence suggests young children accumulate only small amounts of energetic play while attending ECEC. 11 Physical activity intervention research within ECEC services is warranted to positively affect young children’s physical activity while in care. 12, 13 Measuring physical activity levels of children in ECEC is important for evaluating the success of physical activity interventions and determining if services are meeting ACECQA national quality standards. In addition, measuring children’s physical activity can be useful for ECEC services to assess their own progress on physical activity initiatives they may have in place or wish to focus on in the future via their Quality Improvement Plans. Typically, young children’s physical activity levels are measured with direct observation or devices such as accelerometers. 14 However, these methods are more challenging to use in large studies and population-level programs due to their cost, human resourcing requirements, and participant burden. 14 Proxy reports, such as brief assessment tools completed by ECEC educators, could be a useful option for services to simply and quickly report on and monitor progress with children’s physical activity levels. A reliable and valid brief assessment tool could be useful both for research and as a translational tool for ECEC educators. Few tools have been tested, 15-17 especially with regards to measuring children’s energetic play in ECEC specifically. The purpose of this study was to evaluate the inter-rater reliability and concurrent validity of a short educator-reported physical activity assessment tool designed to measure children’s energetic play levels while in ECEC. Methods Participants and setting Participants were recruited from ECEC services participating in the Play Active physical activity policy intervention. The protocol for the trial has previously been described in full. 18 In brief, 81 ECEC services in Perth, Western Australia participated in a pragmatic cluster randomised trial to implement a physical activity policy. Baseline data were collection from January to June 2021, and the policy intervention was implemented from June 2021. Follow up data collection occurred from September 2021 to March 2022. A sub study within the trial was conducted to test the reliability and validity of a short educator-reported assessment of children’s energetic play while in care. A convenience sample of ten participating ECEC services were invited to participate in the sub study, with the aim to recruit 20 children aged two to five years within each of these services. ECEC service directors provided consent for their service to participate in the Play Active trial and parent consent for the sub study was obtained for 156 children across the 10 services. The trial (including sub study) received ethics approval from The University of Western Australia Human Research Ethics Committee (RA/4/20/6120, approved 19/5/2020). A STROBE statement for this cross-sectional sub study is provided in Supplementary File A. Data collection During baseline data collection, educators completed a brief assessment (the Energetic Play Assessment Tool, EPAT) on children’s physical activity levels while in their care. To examine inter-rater reliability, two separate educators were asked to complete the tool for each child. To examine concurrent validity, children were fitted with an accelerometer to collect device-based physical activity. Research staff attended participating services to fit devices on children and provide a study pack for parents consisting of an accelerometer information sheet, accelerometer diary, and a brief socio-demographic survey. Where children did not assent to wearing the device or were not in attendance on the day, devices were sent home in a parent pack. Research staff returned to the services two to three weeks later to pick up devices and parent packs. Reply-paid envelopes were also provided to services to mail back devices returned late. Data collection occurred at baseline, prior to services being notified of their intervention randomisation status. Measures Energetic Play Assessment Tool (EPAT) Educators reported on children’s levels of energetic play using a modified version of the validated Burdette parent proxy report instrument. 19 This tool had previously been modified and validated for use in family child care homes. 15 The tool was adapted for the current study based on feedback and consultation with the Play Active Partner Advisory Group. Namely, the wording of the tool was modified to be specific to energetic play and measured energetic play over four periods (instead of two) over the course of a typical childcare day: arrival to morning tea, morning tea to lunch, lunch to afternoon tea, afternoon tea to departure. In addition, the response options were modified from time intervals (e.g., 1-15 minutes) to a 5-point rating scale: very rarely energetic, rarely energetic, sometimes energetic, often energetic, very often energetic. Energetic play was defined for educators as “Activities that make children huff and puff (e.g., running, jumping, skipping, dancing, riding, climbing and energetic games). It can be spontaneous/unstructured or intentionally planned/structured. It can be done indoors and outdoors.” Each response option was defined with regards to how much energetic play children did in relation to other movement behaviours (light activities and games and sedentary time). For example, ‘often energetic’ was defined as: “Often take part in energetic play (e.g., running, jumping, climbing, dancing, riding bikes or scooters); often do light movements (e.g., standing up doing arts or crafts, playing in a cubby house, pottering around the playground); rarely sit or lie down looking at books, doing puzzles, doing quiet play or learning”. Educators could also record whether the child was sleeping during that period. Accelerometry Objective measurements of children’s physical activity were obtained using ActiGraph GT3X+ accelerometers (ActiGraph Corporation, Pensacola, FL USA) worn on the right hip during waking hours for seven consecutive days. Raw accelerometer data (30 Hz) was processed using a random forest machine learning physical activity classification model for preschool-aged children developed by Ahmadi and colleagues 20 to determine daily time spent being sedentary, light intensity activities and games, walking, running, and moderate-to-vigorous activities and games. Non-wear periods were identified by summing the time periods in which the standard deviation of the accelerometer signal vector magnitude was <13 mg for ≥30 consecutive minutes. 21 The random forest model exhibits significantly higher agreement with measured physical activity intensity than cut-point methods and evidence of equivalence with directly observed time in sedentary activity, light-intensity physical activity, and moderate-to-vigorous physical activity. 22 Energetic play was calculated as the sum of walking, running, and moderate-to-vigorous activities and games. Total physical activity was calculated as the sum of light intensity activities and games and energetic play. For the current study, only data captured during ECEC attendance were included in the analysis. Start and end ECEC attendance times were recorded by parents in an accelerometer diary. Missing attendance times were imputed as 09:30 start and 15:00 finish for known ECEC days. Participant data was included if they contributed at least one ECEC day with accelerometer wear time of at least 75% of attendance time. Child sociodemographic characteristics Parents completed a short survey to report their child’s sex and date of birth, and their own age and level of education. Analysis Inter-rater reliability was assessed separately for each period of the ECEC day using weighted kappa statistics for non-unique raters. Sleep was considered the least active response option (=1) and the five-point rating scale coded from 2 (very rarely energetic) to 6 (very often energetic). In addition, a mean EPAT score was calculated for each child by averaging the responses for all four time periods and inter-rater reliability for the scores were assessed using intra-class correlation coefficients (ICCs) in a one-way random-effects model. Kappa statistics were interpreted using cut points of: <0.00=poor; 0.00-0.20=slight; 0.21-0.40=fair; 0.41-0.60=moderate; 0.61-0.80=substantial; 0.81-1.00=almost perfect. 23 ICCs were interpreted using cut points of: <0.50=poor; 0.50-0.74=moderate; 0.75-0.90=good; and >0.90=excellent. 24 Concurrent validity was assessed by calculating correlations between mean EPAT scores and device-measured physical activity (daily minutes of energetic play and total physical activity) adjusted for device wear time; correlations were calculated for rater one and rater two. Data from eighty children with sociodemographic data and complete EPAT data were included in the analysis; the validity analysis sample comprised n=72 children with accelerometer data. All data analysis were undertaken in Stata v18. 25 A sensitivity analysis excluding sleep was also performed; conclusions were unchanged from the main findings. Results Of the eighty children, 48 were boys (60%) and the median age was 3.4 years (Table 1). Most parents were female (91%) and had a university (bachelor or postgraduate degree) level of education (58%). Across the 10 participating ECEC services, a median of nine children participated (interquartile range=4). Children accumulated a daily average of 22 minutes of energetic play (standard deviation, SD=13.0) and 250 minutes of total physical activity (SD=60.7) while attending ECEC. Mean device wear time was 6.4 hours (SD=1.4). Mean EPAT scores were 4.5 (SD=0.7) for both raters. Table 1. Children’s sociodemographic characteristics and physical activity during ECEC Sample (n=80) Child age, median years (IQR) 3.4 (1.2) n (%) Child gender Female 32 (40.0) Parent gender Female 73 (93.6) Parent education Year 12 or less 12 (15.2) Trade or diploma 21 (26.6) University 46 (58.2) Mean (SD) Device-based physical activity 1 Energetic play (minutes/childcare day) 22.1 (13.0) Total physical activity (minutes/childcare day) 250.7 (60.7) EPAT 2 score Rater 1 3 4.5 (0.7) Rater 2 4 4.5 (0.7) 1 Mean unadjusted device-based physical activity. 2 EPAT=Energetic Play Assessment Tool. 3 Rater 1=mean EPAT score from first educator. 4 Rater 2=mean EPAT score from second educator. Inter-rater reliability Kappa statistics were rated as fair for the morning tea to lunch period (0.386), moderate for arrival to morning tea (0.525) and afternoon tea to departure (0.582), and substantial for lunch to afternoon tea (0.714, Table 2 ). Percentage agreement between raters was around 90% for all four time periods. The ICC for the EPAT score for combined time periods was good at 0.751 (95% CI 0.637, 0.832). Table 2. Inter-rater reliability results for Energetic Play Assessment Tool (n=80) Energetic play rating Kappa statistic (% agreement) ICC (95% CI) Arrival to morning tea 0.525 (90.5) Morning tea to lunch 0.386 (89.5) Lunch to afternoon tea 0.714 (91.5) Afternoon tea to departure 0.582 (91.8) Mean score 0.751 (0.637, 0.832) ICC=intra-class correlation coefficient Concurrent validity Correlations between educator-reported energetic play and children’s device-measured energetic play were low and not statistically significant (rater 1: r =0.009, p =0.940; rater 2: r =0.137, p =0.255). However, educator-reported energetic play scores were significantly and positively correlated with device-measured total physical activity for both raters (rater 1: r =0.237, p =0.046; rater 2: r =0.298, p =0.012). Discussion This study tested the inter-rater reliability and concurrent validity of a short educator-reported physical activity assessment tool designed to measure children’s energetic play levels at ECEC. The brief energetic play assessment tool (EPAT) had moderate evidence of reliability. There was limited evidence of validity when compared with device-measured physical activity, with a weak positive correlation observed for total physical activity but not energetic play. Proxy-report ratings of children’s physical activity levels in childcare settings have been correlated with device-based measures of total physical activity in two previous studies with similarly aged children, one from Japan 16 and one from the United States of America. 15 The latter of these studies also found proxy-report ratings of children’s physical activity correlated with device-measured moderate-to-vigorous intensity physical activity. 15 The lack of correlation between the EPAT and device-measured energetic play in the current study was unexpected despite attempts for the tool to be specific to children’s energetic play, and informed by feedback from the Partner Advisory Group (ECEC providers and professional bodies) as well as ECEC directors and educators. Although the EPAT was not designed to measure total physical activity, there was a positive correlation between EPAT scores and device-measured total physical activity. However, the magnitude of the correlation was small (0.24-0.30). Children’s energetic play may be harder for ECEC educators to assess, as it is typically done in short bursts throughout the day 26 and over a childcare day equates only to approximately 15 to 29 minutes. 11 Staff training on how to identify energetic play could help to overcome this challenge. However, such training would need to be short and easily accessible. High educator turnover, high workloads, and understaffing within the ECEC sector in Australia 27 may also contribute to the challenges of assessing and reporting children’s energetic play during childcare. The brief tool used in this study is short (less than 30 seconds to complete for each child) and easy to use. Compared to direct observation or device-based measures of children’s physical activity, such educator-reported tools can provide cost-savings for researchers and ECEC services. Services can use brief tools like the EPAT to track progress with their Quality Improvement Plans to support and promote children’s physical activity in line with ACECQA National Quality Standards. Limitations The brief assessment tool used in this study requires further development to increase its accuracy to capture children’s energetic play. For instance, time-based response options used in other tools 15 may improve accuracy. In addition, future research should examine the test-retest and intra-rater reliability of the brief energetic play assessment tool in larger samples. Finally, children in this study were from a convenience sample of ECEC services participating in the larger Play Active trial, which could reduce generalisability of the findings. Conclusion The educator-completed brief energetic play assessment tool had moderate evidence of inter-rater reliability, however, weak correlations with device-based physical activity were only observed for total physical activity and not energetic play. While brief tools have a dual-purpose for educators to understand children’s physical activity levels and monitor progress, further research is needed to improve the accuracy of the tool for it to be used for research purposes. Valid and reliable brief assessment tools can help guide physical activity strategies that enable ECEC services to meet—and potentially exceed—national quality standards for supporting and promoting children’s physical activity. Declarations Ethics approval and consent to participate Ethics approval was obtained from The University of Western Australia Human Research Ethics Committee (RA/4/20/6120 approved 19/5/2020). All participating ECEC services and educators provided informed consent. Consent for publication Not applicable. Availability of data and materials The data sets used and/or analysed during the current study are available from the senior author on reasonable request. The Energetic Play Assessment Tool is available on reasonable request to the senior author. Competing interests None to declare. Funding This work was supported by the National Health and Medical Research Council (NHMRC) partnership project grant (#APP1152086) and partially through the Australian Research Council's Centre of Excellence for Children and Families over the Life Course (#CE200100025). HC is supported by The Kids Research Institute Australia Ascend Senior Fellowship. The funding bodies had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript. Authors’ contributions EA: methodology, investigation, data curation, formal analysis, writing – original draft, writing – review and editing, project administration. SGT: conceptualisation, methodology, data curation, formal analysis, writing – review and editing, funding acquisition. HC: conceptualisation, methodology, investigation, writing – original draft, writing – review and editing, supervision, project administration, funding acquisition. All authors read and approved the final manuscript. Acknowledgements Grant investigators (Hayley Christian, Stewart Trost, Michael Rosenberg, Donna Cross, Trevor Shilton, Jasper Schipperijn, Leanne Lester, Georgina Trapp, Ashleigh Thornton, Clover Maitland), partner organisations (Goodstart Early Learning, Minderoo Foundation, CoLab for Kids, Nature Play Australia, Cancer Council Western Australia (WA), Australian Childcare Alliance, WA Department of Local Government, Sport and Cultural Industries and WA Department of Health) and project staff (Emma Adams, Phoebe George, Elizabeth Wenden, Andrea Nathan) are all gratefully acknowledged. This was a joint initiative between the Telethon Kids Institute, University of Western Australia, Queensland University of Technology, University of Southern Denmark, Goodstart Early Learning, Minderoo Foundation-CoLab for Kids, Nature Play Australia, Cancer Council WA, Australian Childcare Alliance, WA Department of Local Government, Sport and Cultural Industries and WA Department of Health References Carson V, Lee E-Y, Hewitt L, Jennings C, Hunter S, Kuzik N, et al. Systematic review of the relationships between physical activity and health indicators in the early years (0-4 years). BMC Public Health. 2017;17(Suppl 5):854-. World Health Organization. Guidelines on physical activity, sedentary behaviour and sleep for children under 5 years of age. Geneva, Switzerland: World Health Organization; 2019. Reilly JJ. Low levels of objectively measured physical activity in preschoolers in child care. Med Sci Sports Exerc. 2010;42(3):502-7. Tucker P. The physical activity levels of preschool-aged children: A systematic review. Early Childhood Research Quarterly. 2008;23(4):547-58. Christian H, Rosenberg M, Trost S, Schipperijn J, Maitland C, Trapp G, et al. A snapshot of the PLAYCE project: Findings from the Western Australian PLAY Spaces and Environments for Children’s Physical Activity Study. Perth, Western Australia: University of Western Australia, School of Population and Global Health; 2018. Christian HE, Adams EK, Moore HL, Nathan A, Murray K, Schipperijn J, et al. Developmental trends in young children’s device-measured physical activity and sedentary behaviour. Int J Behav Nutr Phys Act. 2024;21(1):97. World Health Organization. Prioritizing areas for action in the field of population-based prevention of childhood obesity: a set of tools for Member States to determine and identify priority areas for action. Geneva: World Health Organization; 2012. Department of Education. Child Care in Australia report June quarter 2021: Australian Govermnent; 2022. Available from: https://www.dese.gov.au/child-care-package/early-childhood-data-and-reports/quarterly-reports/child-care-australia-report-june-quarter-2021#toc-child-care-usage. ACECQA. Australian Children's Education & Care Quality Authority: Australian Children's Education & Care Quality Authority; 2022. Available from: https://www.acecqa.gov.au/. Wolfenden L, Finch M, Nathan N, Weaver N, Wiggers J, Yoong SL, et al. Factors associated with early childhood education and care service implementation of healthy eating and physical activity policies and practices in Australia: a cross-sectional study. Transl Behav Med. 2015;5(3):327-34. Trost SG, Schipperijn J, Nathan A, Wolfenden L, Yoong S, Shilton T, et al. Population-referenced percentiles for total movement and energetic play at early childhood education and care. Journal of Science and Medicine in Sport. 2024;27(12):856-62. Christian HE, Cross D, Rosenberg M, Schipperijn J, Shilton T, Trapp G, et al. Development of physical activity policy and implementation strategies for early childhood education and care settings using the Delphi process. Int J Behav Nutr Phys Act. 2020;17(1):131. Bell AC, Finch M, Wolfenden L, Fitzgerald M, Morgan PJ, Jones J, et al. Child physical activity levels and associations with modifiable characteristics in centre-based childcare. Australian and New Zealand Journal of Public Health. 2015;39(3):232-6. Phillips SM, Summerbell C, Hobbs M, Hesketh KR, Saxena S, Muir C, et al. A systematic review of the validity, reliability, and feasibility of measurement tools used to assess the physical activity and sedentary behaviour of pre-school aged children. Int J Behav Nutr Phys Act. 2021;18(1):141. Rice KR, Joschtel B, Trost SG. Validity of family child care providers' proxy reports on children's physical activity. Childhood Obesity. 2013;9(5):393-8. Chen X, Sekine M, Hamanishi S, Wang H, Hayashikawa Y, Yamagami T, et al. The Validity of Nursery Teachers' Report on the Physical Activity of Young Children. Journal of Epidemiology. 2002;12(5):367-74. Tulve NS, Jones PA, McCurdy T, Croghan CW. A Pilot Study Using an Accelerometer to Evaluate a Caregiver's Interpretation of an Infant or Toddler's Activity Level as Recorded in a Time Activity Diary. Research quarterly for exercise and sport. 2007;78(4):375-83. Nathan A, Adams E, Trost S, Cross D, Schipperijn J, McLaughlin M, et al. Evaluating the effectiveness of the Play Active policy intervention and implementation support in early childhood education and care: a pragmatic cluster randomised trial protocol. BMC Public Health. 2022;22(1):306-. Burdette HL, Whitaker RC, Daniels SR. Parental report of outdoor playtime as a measure of physical activity in preschool-aged children. Arch Pediatr Adolesc Med. 2004;158(4):353. Ahmadi MN, Pavey TG, Trost SG. Machine Learning Models for Classifying Physical Activity in Free-Living Preschool Children. Sensors (Basel, Switzerland). 2020;20(16):4364. Ahmadi MN, Nathan N, Sutherland R, Wolfenden L, Trost SG. Non-wear or sleep? Evaluation of five non-wear detection algorithms for raw accelerometer data. Journal of Sports Sciences. 2020;38(4):399-404. Ahmadi MN, Trost SG. Device-based measurement of physical activity in pre-schoolers: Comparison of machine learning and cut point methods. PLOS ONE. 2022;17(4):e0266970. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159-74. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155-63. StataCorp. Stata Statistical Software: Release 18. College Station, TX: StataCorp LLC; 2023. Cliff DP, Reilly JJ, Okely AD. Methodological considerations in using accelerometers to assess habitual physical activity in children aged 0–5 years. J Sci Med Sport. 2009;12(5):557-67. United Workers Union. Exhausted, Undervalued and Leaving: The crisis in Early Education. Australia: United Workers Union; 2021. Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information

Related documents

• Children’s Leisure-Time Activities in the Context of Maternal Education and the Family’s Perceived Economic Situation#1135846
• Nutrition and Soccer for Pediatric Overweight and Obesity#891356
• Decision-Making and the Relationship to Food Intake in Kids#862511
• Physical Literacy through Active plaY and environmental affordances in outdoor kindergarten settings: a multidimensional approach to Unveiling Potential (PLAY UP) - study protocol for a cluster-randomised trial.#368340
• ChildGaze: A Benchmark Dataset for Collaborative Behavior Understanding in Children#973934