Main Body

Chapter 3. Methodology

This study uses Internet-based literature research to explore relevant public policy documents supporting online open e-textbook publishing systems. Text-analysis of documents has been employed in OER research (Mackintosh, 2017; Weller, 2016) and is applicable to analyzing OER policy text in this inquiry. This study re-envisions OER research from a critical theory perspective, employing hermeneutic tools to analyze words and context in institutional OER policy, to interpret relevant policy and orientation with respect to open e-textbook publishing systems in academic institutions. Cohen et al. (2018, p. 49) noted that a critical theory paradigm “links the conduct of educational research with politics and policy making,” just as this inquiry links OER policies to key terms to interpret orientation and implications for education.

Hermeneutics is “the theory and methodology of interpretation, especially of scriptural text” (“The American Heritage Dictionary Entry,” 2020). A philosophical meaning of hermeneutics is “the study and interpretation of human behaviour and social institutions” (“Hermeneutics,” 2021). Although a policy is not a scripture, policy documents are a form of doctrine for educational institutions that contain text for analysis, and hermeneutic tools provide a way to formalize the interpretation of such text. Rockwell and Sinclair (2016, p. 23) asserted that hermeneutica is part of a movement that integrates method and interrogation. According to Vieira and Queiroz (2017, p. 9, as cited in Danner, 2006) “hermeneutic is an art of interpretation of understanding.” Hermeneutic tools are employed in this study to interpret OER policy and dedicated policy for open e-textbook publishing systems in academic institutions.

According to Rockwell (2003, p. 2), text-analysis tools have their roots in concordance, and “concordances are examples of the sorts of “augmentation” tools that extend our scholarly reach and therefore assist in intellectual work.” Rockwell (2003, p. 6) identified three hermeneutical principles central to text-analysis tools:

  • use of a concordance for interpreting a text presumes coherence of the content

  • concordance is a hybrid assemblage of new text

  • concordance is generated systematically by a query process that generates a text in response to a reader’s choice

Rockwell’s (2003, p. 7) hermeneutical principles involve the following computer generated hybrid text qualities of:

  • computing interpretation aides as distinguished from human interpretation

  • analytically processed text for the purposes of discovery and reflection

  • leveraging computing capabilities in quantifications and visualization of multimedia and transmedia

  • no a-priori privilege to certain processes of decomposition and recomposition

Rockwell (2003, p. 7) aims to re-envision concordance with hermeneutic tools that generate hybrid text from a corpus in combination with user’s choices and procedures. Rockwell (2003, p. 8) posits that “a hermeneutic disciplined play postpones questions of principle indefinitely which [sic] another way of keeping them in play.” The advantage of hermeneutics in the context of the research inquiry is the potential for future discourse on open e-textbook publishing system policy.

A policy document can be quantified and visualized to gain insight into its contents, by examining the words that make up the document (Silge & Robinson, 2021). Although there are many text analysis software in existence, Voyant Tools was selected as the best fit for purpose in reading and analysis of text within the context of this exploratory research of OER or dedicated policy documents for open e-textbook publishing systems. Lewins and Silver (2009, pp. 3–4) noted that an appropriate CAQDAS package is one that provides a range tools that will support effective analysis and allow the researcher to control the interpretive process and tool selection.

The Voyant Tools collection of twenty-four tools includes an array of textural-analysis functions within a single application, supported by multi-platform open-source that can generate hybrid results with minimal effort in the form of text, graphics, interactive elements, and HTML embedded dynamic output. Voyant Tools software version 2.0 branch was initially released in 2014 (Sinclair, 2014a, 2014b), and continues development, having evolved from “HyperPo and Taporware and more distantly TACT” (Sinclair & Rockwell, 2016a, sec. Credits). Voyant Tools design principles highlight systematic, adaptive, and reference qualities appropriate to text-analysis of policy documents (Sinclair & Rockwell, 2016a, sec. Design Principles).

Voyant Tools has been used successfully in many forms of research (Hendrigan, 2019b; Hetenyi, Dr. Lengyel, et al., 2019; Philbin, 2018; Rockwell & Sinclair, 2016). Voyant Tools is considered a modern text-analysis environment meant to support agile hermeneutics (Rockwell & Sinclair, 2016, p. 23). Rockwell and Sinclair (2016, p. 96) described agile hermeneutics as a mixture of text representation and analysis interwoven into the resulting research and original work. Although Rockwell and Sinclair (2016) asserted that hermeneutica can provide new views on information, they cautioned that there are limitations such as false positives or the structure of the message which could be misinterpreted by the software. In addition, Voyant Tools has limited language-specific functionality such as parts-of-speech analysis (Sinclair, 2020b), that is available in such applications as the Stanford Log-linear Part-Of-Speech Tagger (The Stanford Natural Language Processing Group, n.d.). Since this study explores patterns and presence of e-textbooks and publishing systems in institutional policy documents, such language-specific functionality (e.g. parts-of-speech) was not employed, but could be a consideration for future text-analysis research. The procedures for data collection in exploring OER policy documents using the Voyant Tools server edition are outlined in Appendix D.

Policy Scope

The selection of the most recent online policy documents for text-analysis is challenged by what distinguishes relevant samples for the research inquiry from less relevant policies. In addition, strategy documents could be considered a form of policy in development (Memorandum of Understanding Open Educational Resources, 2014; University of Saskatchewan OER Strategic Plan 2019-2023, 2019). Skidmore (2019) blended OER policy and practices with open learning, thereby broadening and diluting dedicated OER policy and subsets such as open e-textbook publishing systems. The current literature research revealed that Canadian OER policy and open e-textbook publishing system policy is predominantly absent from academic institutions involved with open e-textbook production. However, table 3.1 outlines the broad scope of relevant documents for the research inquiry.

Table 3.1 Institutional online open e-textbook publishing system policy document relevancy (S = scope; X = outside scope)

Dedicated

X

X

S

OER subset

X

X

S

OER

X

X

S

Private

Public Declaration

Public Policy

The ideal relevant research document is a dedicated institutional policy that articulates the inquiry keywords (see Appendix D section Keywords), such as ‘open e-textbooks’ and corresponding prevalent open e-publishing system that generates the e-textbooks (e.g., Pressbooks). An OER general or subset policy that includes one or more keywords (Appendix D) could be considered a relevant document for inclusion in a Voyant Tools corpus, in the absence of a dedicated open e-textbook policy within an institution. Policy Framework, policy framework policy, and intellectual property rights are outside the scope of the inquiry. In addition, Public declarations of intentions toward OER is considered outside the scope of this inquiry. Private documents are not considered as they are not open to the public Internet and therefore unusable in the context of open access information and this open research. Hence, a minimum criteria for a relevant corpus is a public OER policy document from an institution that has adopted an open e-textbook publishing system.

Internet site selection for policy documents is limited to public Canadian academic institutions engaged with an Internet open e-textbook publishing system to produce open e-textbooks in english or french language. However, the distinction in what constitutes institutional engagement in online open e-textbook publishing systems includes a variation in forms of adoption, such as the AU situation that started as a campus hosted system and subsequently migrated to an off-campus publishing service using the same open-source publishing system (i.e., Pressbooks) as the original on-campus site. However, the current trend across Canada is to aggregate academic institutions onto a central Internet open e-textbook publishing system, such as the inter-organizational partnerships between British Columbia, Ontario, and Manitoba (“BCcampus OpenEd Resources,” n.d.; ECampusOntario Open Library Portal, n.d.; Manitoba Open Textbook Initiative, n.d.). Institutional partnerships exist in Canada such as the SAIT and U of A partnership onto an online open e-textbook publishing system (Norman, 2021; University of Alberta, n.d.-c). More general inter-governmental cooperation initiatives that relate to open e-textbooks is exemplified by the OER memorandum between British Columbia, Alberta, and Saskatchewan (Memorandum of Understanding Open Educational Resources, 2014).

Research Design and Methods

The exploratory design of the study is complimented by agile design, and is guided by the research question, which is framed by a critical theory paradigm. The hermeneutic methods facilitated qualitative and quantitative approaches for close and distant reading (Jänicke et al., 2017) of relevant OER policy documents from Canadian academic institutions from table 3.1. Quantitative analysis (i.e. via text-analysis software) of qualitative data (e.g. text) offers a hybrid interrogation of OER policy documents.

Agile Design. Agile research has been integrated into this exploration as a way finding (i.e., sense making) of emergent ideas for Internet open e-textbook publishing system policy. The continuous cycles of exploring, reflecting, and learning provided an opportunity to improve the development process of the research project on OER policy with few related Canadian research literature. A policy focus emerged from an initial generalized exploration exemplified in Appendix C pilot literature search.

Robson (2011, p. 159) asserted that “researchers using flexible designs do need to seriously concern themselves with the reliability of their methods and research practices,” due to the non-standardization of methods generating qualitative data. Robson’s perspective parallels Labaree’s (2020) assertion that exploratory design lacks rigorous data gathering methods. Robson (2011, p. 159) believed that demonstrating “being thorough, careful and honest in carrying out the research” via full recording minimized risks to validity and reliability. This agile designed research aims to provide an open re-textbook website (https://mem-mem.com/research/) as a form of open-access to the research and use open-source software for reproducibility of results from the same data, with the least digital barriers possible.

Exploratory Design. An exploratory research, supported by agile design, provides flexibility in addressing the research questions presented in Chapter 1 research question section. The research aggregates the following goals of exploratory design, as discussed by Labaree (2020), for producing insights toward the research questions:

  • Extend ken of OER policy to open e-textbook publishing systems.

  • Develop new ideas, hypothesis, and connections between critical theory and open e-textbook publishing system policy discourse.

  • Discern potential directions of future research and techniques with open e-textbook policy area.

  • Refinement of investigation and create new research questions.

  • Alignment with academic social science research (Weimer & Vining, 2004).

The intention of this study is to provide a broader background to the notion of an open e-textbook publishing system policy. This research discourse provides an opportunity to re-examine existing terminology and extend the contemporary concept of an open e-textbook. In addition to gaining insights, this investigation provides methodologies involving open software, data, and information for future research.

Methods. Rockwell & Sinclair (2016, p. 23) argue that hermeneutica is “part of a movement that integrates method and interrogation.” Voyant Tools text-analysis environment facilitates exploratory research design with systematic analysis and increased closeness to the data (Lewins & Silver, 2009). However, Hetenyi, Lengyel, et al. (2019) highlighted the importance of careful, intentional consideration was needed in avoiding falsely interpreting visualized data. The investigation involves collecting a corpus of OER policy texts and following a process of close reading via researcher observations and interpretation of policy documents, and distant reading via Voyant Tools followed by research interpretations of Voyant Tools output. In addition, multiple tools are used for single document corpus that are applicable to multiple document corpus.

Voyant Tools has been considered a viable text-analysis and visualization application for researching a wide range of documents (Daines III et al., 2018; Hendrigan, 2019a; Hetenyi, Lengyel, et al., 2019; Sinclair, 2020b; Steiner et al., 2014). Voyant Tools aligns with the criteria for open source software selection outlined in Chapter 1 scope and limitations section. Voyant Tools is described as “a web-based text reading and analysis environment … to facilitate reading and interpretive practices” (Sinclair & Rockwell, 2016a), and is guided by design principles applicable to the research including modularity, generalization, flexibility, scalability, and reference-ability. Experimental tools in Voyant Tools, such as RezoVi and Veliza, were outside the scope of this research inquiry. In addition, tools such as TextualArc, designed to provide interactive visualization, were not used for the investigation.

Voyant Tools suite provides three main forms of dynamic and static output: textual, graphical, and interactive text with graphics. Although Voyant Tools provides dynamic interactive hyperlink functionality between tool displays, the study limited text analysis output to a static presentation for adaptation to portable publishing document formats (i.e., ODT and PDF). Tool selection for the study was challenged by the number of various tools that could potentially contribute to the interpretation process. The criteria for reducing the toolset to a manageable representative selection of textual, graphical, and mixed text/graphic views within the design parameters of the exploratory study, was to choose tools applicable to a single document corpus that could be extensible to a multi-document corpus within a static presentation context that provided information to aide in interpretation of the selected OER policy documents.

Rockwell et al. (1999, p. 8), asserted that interactive visualization affords users easier “connection between the words used in the statistical analysis and resulting data.” However, interactive tools such as Bubbles and Mandala were not used for the study but could be considered for embedded interactive web-based presentations.

The following minimalist Voyant Tool set for generating text (T) and/or graphic (G) output were chosen based on the aforementioned criteria of a single document corpus focus:

  • Cirrus (G)

  • Contexts (T)

  • Corpus Collocates (T)

  • Corpus Terms (T), and (G) in a multiple document corpus

  • Phrases (T and G)

  • StreamGraph (G)

  • Summary (T)

  • Trends (G)

Due to the plethora of tools in a single text-analysis open source application, it is tempting to indiscriminately choose all the tools. However, beyond the scope of possible interpretations from output by such a toolset is the limiting factor of time for the research inquiry, and thus a discreet selection was chosen, whereas there are opportunities for future research to employ a broader set of tools, particularly the interactive aspects of Voyant Tools. In addition, correlations for the single document focus are considered outside the scope of this inquiry to avoid invalid interpretations, following upon Sinclair and Rockwell’s (2016f) caution on validity with “relatively small number of values.”

Ethics Considerations

The Merriam-Webster dictionary defines ethics as principles of conduct governing an individual or group (Definition of ETHICS, n.d.). According to Cohen, Manion, and Morris (2018), “ethical decisions are contextually situated – socially, politically, institutionally, culturally, personally – and each piece of research raises ethical issues and dilemmas” (p. 261). Therefore, the research website (mem-mem.com/research) aims for open research in development and publishing of the work with a Creative Commons license, and declaration of disclosure and ethics in the front matter. In addition, persons names and titles from email correspondence are removed from the citation manager before Internet publishing. This study is conducted in accordance with the Alberta Freedom of Information and Protection of Privacy Act (Alberta, 2015).

Since this study is an exploratory investigation that is primarily text analysis of online public policy document, there are no anticipated risks to organizations, or academic institutions.


History

Feedback/Errata

Comments are closed.