Article Review : Proposed Model for Evaluating C/LMS Usage in Higher Education Institutions

Critical Review: 

Proposed Model for Evaluating C/LMS Usage in Higher Education Institutions

1.      Introduction

The article under review relates to a proposed model presented in evaluating the usage of content/learning management systems used in a higher educational environment. The citation of the article reviewed is presented as follows.

Janossy, J. & Hover, T. (2008). Proposed Model for Evaluating C/LMS Usage, Proceedings of Society for Information Technology & Teacher Education International Conference 2008 (pp. 2979-2986). Chesapeake, VA: AACE Retrieved from http://www.editlib.org.proxy.lib.sfu.ca/p/27681.

1.1)   Purpose of the Article

 The main goal of this article is to propose a common model to evaluate the use of content/learning management systems (LMS)

  • Across different  institutions
  • Across  different software platforms

In order to make comparisons of the significance of its use for several reasons. They are

  • Administrators across institutions using C/LMS interested to assess the actual extent to which faculty usage of the system. This would help them to justify substantial costs incurred in the acquisition, construction and maintenance of such a system and make future decisions related to the use of LMS.
  • To make comparison between institutions of the use of the system as an administrative tool vs. teaching tool
  • To assess operating cost per student in running such a system and to make inter institutional comparisons.

1.2)    The Intended Audience

Based on the contents, the primary audience that this article seems to relate to would be the administrators of institutions in general. As a secondary audience, benefits to faculty members of such a model have been sparsely mentioned.

1.3)    Research Question Addressed

The article does not introduce a specific research question. But based on the contents, the main research question seems to revolve around “How best learning management systems could be compared across different institutions using different platforms?” In the discussion section, the author presents several other research questions that the intended users could use in applying this model to their intuitions.

1.4)    The Thesis Statement

While the thesis statement is not clearly highlighted, the main premise of the author in presenting this model seems to be that all institutions would benefit from having a common model to evaluate the usage of its LMS. The objective of this seems to be for this model to help allow inter-institutional comparisons to make administrative decisions. The author seems to believe that this one size fit model will benefit institutions by sharing common information. Also in the introduction, the author starts off by saying “although different learning management systems are used by different organizations, they are all build on a common model and it makes sense to compare the common elements of all these models”.

1.5)  Design of the Study/Model  

A brief literature review is presented with some of the previous work related to this area. With this, the author has gone directly into the proposed model. The author has not related how the proposed model seems to fulfil any gaps in current literature. The contents of the model seem to fulfil the purposes stated in section A and not based on any previous work.

Usage   Level

Intensity   Stage

Label   for the intensity stage

Description

Level 0No usage of the C/LMS

0

Instructor does not activate and use a LMS  Each of these   levels are clearly defined
Level IDocument Distribution

1

Minimal document provision site

2

Minimal document provision – slightly more productive

3

Minimal document  provision – visually appealing

4

Competent document provision site
Level IIWork submission,   feedback

5

Minimal student work turnaround site

6

Adequate student work turnaround site

7

Informative student work turn around site

8

Highly informative work turn around site
Level IIIWork submission,   feedback

9

Highly meaningful turnaround site with electronic testing

10

Highly meaningful turnaround site with asynchronous online discussions
Level IVRecoding for review,   real time distance participation

11

Highly meaningful turnaround…. with web based one way viewing

12

Highly meaningful turnaround…. reviewing via recording at later times

13

Highly   meaningful turnaround…. interactive viewing and participating

1.6)    Empirical Validation/Research 

The above model has not been subjected to any empirical study.  The rationale provided for defining different levels and intensity seems to be based on the author’s experience with learning management systems rather than on any feedback from users. The author seem to have reviewed the “Blackboard” LMS platform in suggesting these intensity levels for all learning management systems. The other LMS platforms referred by the author are WebCT(2007), Desire2Learn (2007), Angel (2007), open source systems such as Sakai (2007) and Moodle (2007). These have only been referred to but not been included in developing the said model.

2.      Analysis  

 2.1 Strengths of the Article

This article presents a few very valuable insights to a researcher interested in this area. They are

  • This is one of the few evaluation tools that clearly categorize the usage of an LMS into clear intensity  levels and then based on that, broader levels of use. The model presented seems to be very simple, clear, logical and practical.
  • A user who is subjected to this model will be able to understand the level of significance they place in using an LMS for teaching purposes. It also will clearly help the user to identify whether they use the LMS as an administrative tool or as a teaching tool of their course/program.
  • This model could be used by administrators in order to assess how their LMS are used by faculty members in order to substantiate any investments and continue resource commitments towards upgrading or maintaining the system.
  • Also the common platform used will help make comparisons across institutions as intended by this model.
  • Professional development centres,  instructional learning design centres could use these findings to develop further pedagogical methodologies to be used to support online or mix mode methods.
  • The questions that the author has raised at the end of the article will help the reader to ask these in order to assess their use of an LMS after subjecting themselves to this assessment. With this addition, this model will be beyond a theoretical framework and could be used as a practical application tool.

2.2 Weakness of the Article

The following could be identified as key weakness associated with the article.

  •  The primary objective of presenting this common evaluation platform is to help administrators within institutions to make comparisons of the usage intensity of the LMS. The author could have built this model as an evaluation tool to support the pedagogical value of technology usage for teaching than using it as a mere administrative tool.
  • The design of the model is not based on any theoretical framework but on a mere practical usage application. While this keeps the model simple to apply, the academic value of this model could be questioned.
  • Not subjecting the model to an empirical study could be identified as a big weakness. The validly of the model needs to be tested if it is to be applied as an evaluation tool across institutions where results are to be used for decision making purposes.
  • The ability of this one size fit model to be used across all institutions applying to different platforms to compare their usage into one model may be questioned. Although the basic underlining theories of many learning management systems are common, the vast difference of applications used will not allow this common model to be applied for different platforms.

2.3 Critical Comments  

The following set of points will look at different aspects of the article critically.

  •  The goals and the purpose stated for the model development seems to be defined quite narrowly. As  highlighted under weakness, the pedagogical value of this model to assess the      sage of technology for teaching would be far more superior than using this as an administrative tool. If this perspective was considered, the model could have evaluated different learning objectives and their associated technology applications making it a very useful formative and summative evaluation tool. If the latter was to be considered, then a new research question could be raised such as “How best could a common evaluation tool be developed to assess the learning objectives of a given course/program      delivered through a learning management system?’. The solution to this kind of a question would add far more benefits than using this for inter institutional comparisons.
  • If the above was considered, then the intended audience of this report could be further expanded to faculty members, instructional techniques developing bodies, LMS development organizations making this model a widely applicable one.
  • In this case, the thesis statement  of this study would be to develop a technology based learning methodology  evaluation tool for faculty members.
  • If the model was tested, the validity of it could be verified leading to a greater level of acceptance among the academic community.

2.4 Main Points Learned Through the Article

The main points reflected in the article highlighted the importance of using a common platform to evaluate the usage of LMS for administrative purposes. However, the reviewer seems to suggest that this model has a higher pedagogical value than using it mere as an administrative tool. Different degrees associated with intensity and level of usage could be linked to learning outcomes for a course and this model could act as a design tool as well as an evaluation tool. An empirical study using this model over a period of time could provide valuable insights to designers of future learning management systems to incorporate applications to support a higher level of usage of their systems to deliver courses/programs. I would encourage different users of LMS to subject their usage to this model to assess where they stand in terms of usage. With this, they could use this to support their future course designs using LMS. The findings of this study will also in no doubt provide administrators an understanding of current usage levels, enable them to take corrective action to increase its usage, provide training and professional development or even change the existing LMS to meet organizational and teaching objectives.

3.      Conclusion

In this review, the introduction summarized the purpose, the intended audience, the assumed research question, the suggested thesis statement, the design of the study and comments about the empirical validity of the model. In the analysis section, the main strengthens, weakness, together with critical comments and the main points learned through the article was highlighted. As an overall conclusion, this article is highlighting a very strong conceptual framework which has far more pedagogical value than it is identified for. However, the development and the testing of the model has been executed rather weakly leading the intended user’s to question its validity.  As a future research opportunity, this model could be tested for its usage and proven for its validity not only as an administratively tool but also as a teaching tool helping faculty members to assess their use of technology for pedagogical applications of their courses and programs.

Comments are closed.

%d bloggers like this: