A Tool for Adaptive E-Assessment of Project Management Competences

Chapter 9
A Tool for Adaptive E-Assessment of Project Management Competences

Constanta-Nicoleta Bodea
Academy of Economic Studies, Romania
Maria-Iuliana Dascalu
Academy of Economic Studies, Romania

ABSTRACT
This chapter proposes an e-assessment method for project management competences, using the computer adaptive testing (CAT) principle. Competences are represented using concept space graphs. The proposed model increases the tests configurability by considering several knowledge constraints when an item is selected. The proposed model is also seen as a self-directed-learning-tool, useful in the preparation process for project management certifications. The model is validated by comparison with an existing e-assessment tool, used for simulation purposes; statistic results are presented and analyzed. Although the initial level of knowledge of each user has a great impact on the final results obtained by that user, preparation with the proposed e-assessment method proved to be more efficient.

A Tool for Adaptive E-Assessment of Project Management Competences
held in companies and organizations (Marimuthu & al., 2009), and by the introduction of various tools meant to develop employees’ skills or to fill knowledge gaps (Delcea & Dascalu, 2009). The selection of suitable learning instruments for competence development has, without doubt, had a tremendous impact on both corporate and personal well-being (Kröll, 2007), (Fabra & Camisón, 2009). As a consequence of the technological boom, e-learning is gaining increasing numbers of followers. Although e-learning has overcome time and place constrains (Phobun & Vicheanpanya, 2010), the need for personalized e-learning services appeared: learning performance can be significantly enhanced if individual goals, level of knowledge, background, or learning capabilities are taken into account (Beldaglia &Adiguzela, 2010). Thus, adaptive e-learning instruments represent an important advance in learning system development. Although less complex than the “one-size-fits-all” static approach, adaptive e-learning applications are characterized by greater flexibility (Beldaglia &Adiguzela, 2010) and a more interactive experience (Barla, Bieliková, Bou Ezzeddinne, Kramár, Simko, & Vozár, 2010). Two other modern learning instruments are the personal academic environment (Casanova, Holmes & Huet, 2009) and the adaptive web-based system (Dagger, Wade & Conlan, 2004).
The authors argue that an e-assessment application could be an efficient learning tool in a knowledge society (Bodea & Dascalu, 2009). This statement complies with the opinion of other researchers, who argue that ‘learning from errors is a promising method’, resembling problem-based or task-based education methods (Stergiou & al., 2009): “… every error includes the chance to learn from it, if learners can view it as learning occasion. Therefore, learners have to identify the error and understand the correct solution by comparing the incorrect solution systematically with the correct one. The tutor’s feedback is of great importance” (Stergiou, Georgoulakis, Margari, Aninos, Stamataki, & Stergiou, 2009). In order to increase the learning dimension of an e-assessment application, the authors consider implementing adaptive behaviour, where adaptive is defined as “a capability to change when necessary in order to deal with different situations” (Beldaglia &Adiguzela, 2010).
The current study proposes an e-assessment method for project management competences. Projects, as common organizational structures in knowledge society, greatly require the development of project management competences: ‘in project management, competence development is one of the critical success factors’ (Suikki & al., 2006). The proposed method allows the development of e-assessment applications that serve not only as evaluation tools, but also as self-directed learning instruments. Two strong features of the proposed method are adaptability and orientation towards competences. In order to adapt to the user’s knowledge level, the principle of Computer Adaptive Testing (CAT) is employed.
The chapter has the following structure: after a brief introduction, the research is framed in the corpus of relevant literature regarding computer adaptive testing. The research objective is highlighted and the research methodology is presented, starting from conceptual modeling to the description of the actual computer adaptive testing software solution. In order to check the efficiency of the proposed e-testing solution, a validation experiment is designed, based on the method involving a control group which used a different e-testing solution. The results of the experiment are discussed and finally, future directions and conclusions of the research are described.
RESEARCH CONTEXT
A form of computer based testing which increases flexibility and offers more information about the examinees’ competences is Computer Adaptive Testing (CAT). The principle behind CAT is to adjust the test items’ characteristics to the examinee’s
135
A Tool for Adaptive E-Assessment of Project Management Competences
ability level (Desmarais & Pu, 2005). Basically, if the examinee answers a question correctly, then the next question will be more difficult. If the examinee answers incorrectly, then the next question will be easier. Due to the Item Response Theory, similar scores are obtained by resolving different sets of test items.
CAT brings numerous advantages: it saves up to 50% in test taking and test management time and enhances the test users’ motivation by quickly identifying their needs and competences. CAT also incorporates an instant procedure for computing grades. Consequently, the examinees receive immediate feed-back. Also, the questions adapt to the test taker’s level of knowledge, the testing time is reduced (the test itself is shorter), the uniqueness of test questions is ensured, and the evaluation is more accurate. Further advantages are that the tutors spend less time on test creation, test security is enhanced, supervision concerns are greatly diminished, and students’ performance over time can be easily tracked. But, above all, CAT can be seen as a tool for self-directed learning: a CAT user isn’t tested with the same questions multiple times, receives new challenges, and accurately pinpoints the knowledge gaps. In such a testing environment, the knowledge gained depends on one’s own actions: “CAT tests can be better experiences”, as Linacre concluded in 2000.
CAT has numerous large-scale applications (van der Linden & Glass, 2010):
• PROMIS (Patient-Reported Outcomes Measurement Information System), sponsored by the US National Institute of Health, is used in a medical context to measure depression and anxiety: “The aim of this network is to develop a large bank of items that measures patient-reported outcomes and to create a computerized adaptive testing system that allows for efficient assessment in clinical research of a wide range of chronic diseases.” (van der Linden & Glass, 2010);
• MATHCAT, an adaptive testing system used in the Netherlands, for adult education has two purposes: one is placing examinees into arithmetic/ mathematics courses at three available levels, and the other one is monitoring the students’ achievements. Some of the benefits brought by MATHCAT are as follows: immediate test scoring, feedback mechanisms, and prevention of testing material disclosure (van der Linden & Glass, 2010);
• GMAT (Graduate Management Admission Test), “a standardized assessment intended to help business schools assess the qualifications of applicants for advanced study in business and management” (van der Linden & Glass, 2010); since more than 200,000 examinees take the examination annually, GMAT is among the most wide-spread CAT applications;
• the uniform CPA exam, an exam employed by the American Institute of Certified Public Accountants in licensing certified public accountants. It is a 14-hour test consisting of four sections: auditing and attestation; financial accounting and reporting; regulation; business environment and concepts;
• CASEC (Computerized Assessment System for English Communication), developed by the Japan Institute for Educational Measurement (JIEM), has the following features: adaptation of the items’ difficulty to examinees’ proficiency, online services, immediate feedback, application of item response theory (IRT) in order to assess proficiency in English on a common scale, short testing time, capacity of measuring a wide range of proficiency levels (van der Linden & Glass, 2010);
Although based on a simple idea, CAT is a complex mechanism. Each of the five CAT components discovered by Weiss & Kingsbury in 1984
136
A Tool for Adaptive E-Assessment of Project Management Competences
(Rudner, 1998) has been perfected and tailored to different areas of knowledge evaluation. The five components are: item pool, starting point of the test, item selection algorithm, scoring procedure, and termination criterion.
The item pool represents a collection of questions: the bigger the item pool is, the more accurate the assessments are. Weiss indicated that a pool of 150 – 200 questions, having a balanced distributed level of ability, should suffice for a correct evaluation (Rudner, 1998).
The starting point of a test is given by an initial level of examinee’s ability: it can be taken from user profile information in an e-learning platform, from public evaluation grids or it can be established by the assessor according to pursued interests.
The item selection algorithm establishes rules for selecting the next question. Several methods were proposed and applied, but usually the selected question is the one that requires the closest ability to the examinee’s ability level which has already been guessed by the CAT mechanism, as the Item Response Theory (IRT) suggests (Bjorner et al., 2007; Lee et al., 2008). This means that the selected question contains the biggest amount of information. Information for question i of an examinee can be quantified using the equation (1) (Lee et al., 2008):

iiiiPPP()’()()[()]=−21 (1)
where Q represents the ability (grade, score) already demonstrated by examinee j, and Pi represents the probability of a correct answer to question i; this probability is calculated using the equation (2):
P
uabcccabiiiiiiii(/,,,)exp[.**()]==+−+−−11117 (2)
where Q represents the ability (grade, score) demonstrated by the examinee till question i, airepresents item discrimination, birepresents item difficulty, and cirepresents the guessing parameter. Usually, these characteristics are provided by experts.
The scoring procedure implies updating the examinee ability level (Qfrom equation 1), after each answered question:

iiiiiiiiiuPPPP+=+−−11()’()() (3)
The termination criterion of a test depends on the test creator’s wishes. A test can be stopped when one of the following states is reached: the item pool is exhausted, a time limit is reached, a specific number of questions are selected, a targeted amount of knowledge is verified, and the threshold accuracy is reached.
Each of the five components in a CAT mechanism can be modified, taking into consideration the topic of the test, the knowledge representation of the topic, the competences targeted for review or development, or the test conception reason. For example, Al-A’li improved the CAT mechanism for diminishing the number of questions in a test and, thus, the test length (Al-A’li, 2007). Tao, Wu and Chang developed a CAT variant suitable for daily routines and small-scale scenarios (Tao et al., 2008). Eggen and Straetmans used CAT to establish categories of students (Eggen, & Straetmans, 1996).
RESEARCH OBJECTIVE
The current study proposes a solution for a project management e-assessment application: the application is competences-oriented and adaptable to each user’s knowledge level. The direct use
137
A Tool for Adaptive E-Assessment of Project Management Competences
of the tool is presented: the improvement of the certification process held by the Romanian Association of Project Management. Indirect benefits of using the tool are also suggested: increased performance experienced by firms and users. The logic of the application respects the requirements of the International Project Management Association Standard (ICB, 2008). The CAT components described in the previous section are re-used and adapted to our situation.
CONCEPTUAL MODELING
During the development process of the tool, two stringent problems in conceiving computer tests were taken into consideration:
• creating the questions (conceptual units of the tests), which must comply with certain constraints of complexity, clarity, and relevance, depending on the knowledge domain;
• creating the tests (the algorithms for selecting the questions);
For the first issue, the solution considered to be viable is the one of semantic networks. In order to respect a standardized knowledge structure of the project management domain, the IPMA Competences Baseline has been used (International Project Management Association ICB, 2008). According to IPMA, the project management competences have three components: knowledge, skills and personal attitude. The knowledge component refers to what are the generally accepted practices of project management applied to specific technical disciplines. Skills refer to the capability of applying knowledge in an efficient, effective, professional, and successful manner. Personal attitude implies the commitment to perform in an appropriate and acceptable professional and ethical manner. IPMA competences can be grouped in three categories:
• Technical competencies of delivering projects in a structured way, including the project management process.
• Contextual competencies in managing relations with projects within organisations, programmes and portfolios, based on the knowledge of project characteristics, projects in the organizational context, and project environment.
• Behavioural competences for a positive, collective, and dynamic thrust in nurturing project management professionalism such as leadership, communication, results-orientation, ethics, negotiation, and so forth.
For the second problem, related to test algorithms, a test delivery model has been developed, based on computer adaptive testing (CAT) principle. The principle behind CAT is to adjust the characteristics of test items to the examinee’s ability level (Desmarais & Pu, 2005). Due to the Item Response Theory, similar scores are obtained by resolving different sets of test items. The way in which test questions are managed and the test algorithms themselves are further discussed.
Semantic Networks to Manage Test Questions
Test questions are built using the ontology proposed by the SinPers system (Bodea, 2007). SinPers is an e-learning platform which models digital content with learning objects, according to a predefined domain ontology. Metadata is used to describe the properties of learning objects. Learning objects are explained or assessed by concepts defined in domain ontology. Relationships between concepts can be of three types: Has_part (defining hierarchical relationships), Requires (logical constrains defining the mandatory learning order of the concepts) and Suggested_Order (optionally). Concepts are grouped into 46 competences (technical, behavioral and contextual) as proposed in ICB v3.0, the standard
138
A Tool for Adaptive E-Assessment of Project Management Competences
of the International Project Management Association (see an example in Table 1).In order to obtain tests suitable for certification exams of IPMA level A, B, C or D in project management, tests should verify certain competences. The area of concepts is different from level to level, for the same competence. To find the correct one, concept space graphs are used (Bodea, 2008).
The parameters used to select proper questions from the database are concept lists, obtained from the linearization of concept space graphs. These concept space graphs are extracted from the course ontology (see Figure 1).Each competence from the course ontology has attached, as a parameter, an interval of values (threshold interval). According to this, the competence defines a set of concepts or projects a concept space graph. For this purpose, the path weight is used, as described by Hardas (2006).
In order to define relationships between competences, difficulty levels and threshold intervals, four types of operations are required:
• extraction of the semantic net for a certain element of competence (e.g. C1.19 – Start up) from the ontology; the semantic nets are networks which represents semantic relations (represented by links) between concepts (represented by nodes);
• transformation of the newly created semantic net into a concept space graph, using self-weighted values of the concept nodes (the red numbers in Figure 2), prerequisite weights (blue numbers), link weights (link labels); a concept space graph TCL(,)is a projection of a semantic net with vertices C and links Lwhere each vertex represents a concept and each link with weight lij(,)represents the semantics that concept cj is a prerequisite for learning ci, where (,)ccCijÎand the relative importance of learning cj for learning ci is given by the weight; each vertex in T is further labeled with Ws(self-weight value, which represents the
Table 1. Concepts required by C1.19: “Start-up” competence in the project management domain
Concept Code
Concept Description
DDI
Decision of making the investment
DIP
Document for initiating the project
PRO
Project proposal
CPR
Project charter
DDP
Decision to start the project
EEP
Pre-evaluation of the project
ATP
Assigning the project
Figure 1. Fragment of educational ontology (the part 1: “project”)
139
A Tool for Adaptive E-Assessment of Project Management Competences
relative semantic importance of the root topic itself with respect to all other prerequisites) and Wp(prerequisite-weight value, which represents the cumulative, relative semantic importance of the prerequisite topics to the root node) (Hardas, 2006);
• projection of concept space graph to sub-graphs of different semantic dimension, according to the relationship between nodes path weights (h) and the threshold coefficient (l) ; the following formula can be used:
h
(,)()(,)*().xxWxlxxWxtstmmpmmt0111=−−= (4)
Where nodes x0 and xt are connected to a path given by the set [,...,,...,],xxxxommt+1;
• analysis of different projection sub-graphs, according to the desired coverage of the initial projection graph, in order to obtain the relationship: competence element, level and threshold interval; e.g. “Start up project” competence element, IPMA level D certification requires a coverage threshold of maximum 0.04 (meaning, examinee should know ARP, DDP, DIP, DDO, PRO concepts) and the other levels require a coverage threshold of minimum 0.04 (meaning that the examinee should know ARP, DDP, DIP, DDO, PRO, CPR, EEP concepts); the greater the threshold coefficient is, the smaller the sub-graph area and, of course, the level of required competence will be; the rating for establishing what concepts should be contained by questions, when the purpose of a test is to check “Start-up” knowledge and the test level is D, is described in Table 2.Competences Based Computer Adaptive Testing
Computer Adaptive Testing is based on Item Response Theory, but it can be adapted and optimized according to each situation or field of activity. In a competences-based environment, the target of creating tests for project management knowledge is not only to adapt the questions to the user’s level of ability, but also to a certain area of competences. The success of an adaptive testing application depends on test specifications, item selection algorithm, pool design and rotation, ability estimation, item analysis, and database design (van der Linden, 2010).
Three elements are improved from the classical CAT:
• the item pool is parameterized: the parameters used to select certain questions
Table 2. The coverage analysis for C1.19-“start-up” competence using node path weight in project management automatic assessment
Concept
ŋ
ŋ>=λ (λ=0.01)
ŋ>=λ(λ=0.04)
ARP
0.15
TRUE
TRUE
DDP
0.125
TRUE
TRUE
DIP
0.1
TRUE
TRUE
DDI
0.06
TRUE
TRUE
PRO
0.045
TRUE
TRUE
CPR
0.03
TRUE
FALSE
EEP
0.03
TRUE
FALSE
140
A Tool for Adaptive E-Assessment of Project Management Competences
from database are concept lists, obtained from concept space graphs linearization (see Figure 2);
• termination criteria: the stop criterion is fired when all the concepts needed to be assimilated are among the already tested ones or when the examinee’s ability level (or the score) is lower than 50/100;
• item selection algorithm, which is shown in Figure 3.
The question selected is the one with the maximum information value (equation 1). Because the difficulty parameter (bi from equation 2) is calculated using knowledge structures (it is not provided by experts, as in a classical model), the information function is named knowledge function. Concept space graphs are used to calculate question difficulty: in this case, the graph root is a question, and not a competence, as in Figure 2. The question difficulty is given by the coverage indicator of a projected graph (Hardas, 2006). The selection algorithm is the core of the CAT mechanism, which is shown in Figure 3.We define the following function, which represents the questions set:
q
sqsQQQQQQQQnnn:,(,…,;){,…,,}.,,=+12121 (5)
The concepts needed to answer question q are described in equation (6.1) and the concepts needed to successfully complete assessment T, are described in (6.2).
C
CCCqqqnq=[,,...].12 (6.1)
C
CCCTTTmT=[,,...].12 (6.2)
The stop criterion is fired when all the concepts needed to be assimilated are among the already tested ones or when ability level (or the score) of the examinee is lower than 50/100. We should mention that the initial level of ability (start criteria) is 80. The stop criterion is modeled in equation (7):
(
)(),,||||CCScoreiqsTqiqsi==1501 (7)
SOFTWARE SOLUTION
The e-assessment application (see Figure 4) has service-oriented architecture (see Figure 5).
Figure 2. Knowledge structures in the project management automatic assessment, for C1.19: “start-up” competence
141
A Tool for Adaptive E-Assessment of Project Management Competences
Figure 3. CAT mechanism for project management assessment
Figure 4. The e-assessment application for project management competences
Figure 5. System architecture for the e-assessment application
142
A Tool for Adaptive E-Assessment of Project Management Competences
As shown in Figure 5, the web service which takes the user requests in a formalized way, can call/use three different services:
• item selectors, designated to select proper questions from distributed item pools;
• statistic providers, which compute scores, offer correct answers, and make reports;
• profile loaders, which load information about certain classes of users; there are 3 classes of users: demo users, registered users (the ones who create accounts on infunction
loadQuestions(data)
{
// xml parse
/*<question id=”2” text=”Abordarea orientata pe proiecte reprezinta:” link=””>
<answer code=”A” text=”Modelarea prin intermediul notiunii de proiect a tuturor proceselor de afaceri”/>
<answer code=”B” text=”Aplicarea exclusiva a metodelor, instrumentelor si tehnicilor de managementul proiectelor in procesele de afaceri, in special cele cu grad ridicat de unicitate”/>
<answer code=”C” text=”Aplicarea metodelor, instrumentelor si tehnicilor de managementul proiectelor in procesele de afaceri, in special cele cu grad ridicat de unicitate in vederea imbunatatirii lor”/>
<answer code=”D” text=”Organizarea proiectelor”/>
<answer code=”E” text=”Definirea rolurilor, responsabilitatilor si autoritatilor in proiect”/>
</question>*/
var questions = data.documentElement.getElementsByTagName(“question”);
for (var i = 0; i < questions.length; i++)
{
var crtQs = new CQuestion (i+1, questions[i].getAttribute(“id”),
questions[i].getAttribute(“text”), questions[i].getAttribute(“link”), “false”);
var ans= new Array();
var answers = questions[i].getElementsByTagName(“answer”);
for (var j = 0; j < answers.length; j++)
{
var crtAnswer = new CAnswer (answers[j].getAttribute(“code”), answers[j].getAttribute(“text”), 1, “false”);
ans.push(crtAnswer);
}
crtQs.answers = ans;
g_questions.push(crtQs);
}
}
143
A Tool for Adaptive E-Assessment of Project Management Competences
ternet, with limited rights), and super users (the ones with authorization to the certification preparation process);
The user interface is developed using DHTML, JavaScript and Ajax functions offered by the Dojo library. The test items and the algorithms from the item selectors are the main elements of the e-assessment application. The items are formalized using metadata. The metadata exploitation is made with dynamic HTML, as seen in the code function loadQuestions(data). The final result is a list structure, g_questions, containing the items from a test.The considered adaptive e-testing application is developed on a knowledge base rendered to a database (see Figure 6). This explains why a question’s difficulty is calculated starting from domain concepts, which are mapped to competences in domain ontology.VALIDATION AND COMPARISON WITH OTHER APPLICATIONS
In order to prove the efficiency of the proposed application, a comparative analysis between it and the previous e-testing solution used by the Romanian Association of Project Management is provided. The adaptive e-testing application follows the same interface rules and uses the same questions as the previous one. After registering to the appropriate account, the user receives one question at a time. This is a common feature of adaptive computer testing interfaces: the next question isn’t known when test begins; it depends on the knowledge function of all unselected questions (including the current one), as described in the previous chapter. The CAT user cannot navigate through question screens: one can only go forward. In this way, the system cannot be tricked, as: the answer cannot be changed. Technically, in the classical application, there is only one request to the server for obtaining the test questions. In an adaptive test, the number of requests depends on the number of questions. Consulting equation 7, which highlights the stop criteria, it is readily observable that the number of questions and, implicitly, the duration of the test depend on user competences. In CAT, the overall time is somewhat shortened by test results calculation. In the classical variant, an algorithm is used for calculating the final score and more database queries are
Figure 6. Data structures in the project management adaptive e-testing
144
A Tool for Adaptive E-Assessment of Project Management Competences
necessary. In adaptive testing, score calculations are included in the item selection algorithm.
Data Collection Method
Two groups of 75 master students were separately asked to prepare themselves for the certification exam in project management, level D. The subjects’ description is available in Table 3. All students were initially tested using a pen- and- pencil test, consisting of the same questions. The preparation process lasted 3 months. The first group was allowed to use the adaptive e-testing application 3 times for simulation purposes. The other group was allowed to use the classical e-testing application 3 times. Their simulation results were registered and analyzed. After the preparation period, they all gave the certification exam. The performance evolution was analyzed, for both groups. In the end, the impact of CAT simulation to final results was also quantified.Data Analysis
The simulation results are reflected in Figure 7: each line represents the evolution of one student’s grades. The grades could have been between 0 and 100. In the case of CAT, the general trend is ascending: so, all the students improved their skills using the CAT application. Still, it can be noticed that the ones having a high score at the first simulation didn’t make a significant progress. The explanation is obvious: their questions were very difficult, as a result of their high level of knowledge. Based on this observation, CAT efficiency might decrease as the examinee participates for a higher level of certification. On the other
Figure 7. Successive project management evaluation results in certification simulations
Table 3. Data about students
Criterion
Students who used CAT to prepare themselves
Students who didn’t use CAT to prepare themselves
Age
under 25: 48
between 25 and 35: 23
over 35: 4
under 25: 44
between 25 and 35: 24
over 35: 7
Previous studies (graduated college)
Technology: 35
Business administration & economics: 32
Other: 8
Technology: 29
Business administration & economics: 30
Other: 16
Hours spent daily in front of a computer
Less than 1 h: 0
Between 1 h and 4 h: 27
Between 4 h and 8 h: 42
Over 8 h: 6
Less than 1 h: 1
Between 1 h and 4 h: 33
Between 4 h and 8 h: 40
Over 8 h: 1
145
A Tool for Adaptive E-Assessment of Project Management Competences
hand, on the right hand side of the Figure 7, the results of the classical e-testing simulations are represented. Only some of the participants had an ascending evolution. The trend is still ascending, but the slope is not as steep as that of the students who used CAT.A comparison was made between the final results obtained by the master students who prepared with CAT and the other group of examinees, who prepared with the classical simulation software. Both groups tookthe final exam on the same application (which doesn’t yet implement CAT) and on the same database of questions. The average grade of both groups was similar, but more extreme values were obtained in the second group (the one who didn’t use CAT in preparation). The percentage of failure was much higher in the group who prepared with the classical e-testing tool (see Figure 8).Still, the better results obtained among students who prepared with the CAT software may be due to other factors: for example, the self-efficacy of the computer tests users could be affected by their computer level or gender (İşman & Çelikli, 2008). Because of this, a statistical study was made to establish the influence of students’ initial level of knowledge and the simulation tests taken with the CAT software on their final results. The study was made with the EViews tool. In order to homogenize the initial data, a logarithm was applied. Some other reasons for applying a logarithm were: the residual variable wasn’t normally distributed and exogenous variables weren’t independent, there was a negative correlation of -2.73 between them. Ordinary Least Squares (OLS) method was used: OLS is a well-known method for estimating the missing parameters in linear regression models. The t statistics proves that both initial level of knowledge and CAT simulation scores have an impact on the final test results (see Table 4): a 1.36% growth in initial knowledge level will conduct to 1% growth in final grade and a 0.39% growth in CAT simulation effects will conduct to 1% growth in final grade.According to the R square value (0.8), the endogenous variables are explained in a proportion of 80% by the exogenous ones, so the model seems accurate enough: there is a strong correlation between the final results and the grade obtained in the CAT simulation or the initial
Figure 8. Results obtained in project management certification exams
Table 4. Initial level of knowledge and CAT effects on final examination results: statistical data
Variable
Coefficient
Std. Error
t-Statistic
Prob.
C
-1.636626
0.529571
-3.090476
0.0026
CAT_LEVEL
0.386338
0.082885
4.661126
0.0000
INIT_LEVEL
1.354864
0.120624
11.23215
0.0000
146
A Tool for Adaptive E-Assessment of Project Management Competences
level of knowledge The Wald statistic test came to strengthen the idea that both the considered variables have an impact on final results: the Wald test is a parametric statistical test used when a statistical model with parameters (a linear regression model, in our case) is estimated from a data sample. Wald statistics can be used to test the value of the parameters. Although the impact of the initial level of knowledge is greater, the impact of CAT tests cannot be denied.
FUTURE DEVELOPMENTS
In the future, new experiments to improve the mechanism will be made on the questions database used by simulation certification exams of the Romanian International Project Management Association (http://www.pm.org.ro/certexam/). On the other hand, we want to give the test administrators the possibility to attach other domain ontology, thus also extending the application to other fields of activity.
For a better validation, the data about the students shown in Table 3 will be analyzed and we will try to establish whether the better results obtained in the case of CAT may be due to their age, their computer skills (reflected in the number of hours spent daily in front of a computer) or to their previous studies. At first glance, this information seems similar enough in what regards the two groups. Their computer skills cannot bear a significant influence on their final results: if we compared a pen-and-paper group to a computer-users group, then this information would have been more valuable. Their previous studies are reflected in their initial level of knowledge in project management.
CONCLUSION
A software product to be used for assessing project management competences of within the IPMA certification should comply with the current needs for developing competences. In the new economy, the individual competences are regarded as welfare generators as demonstrated using the Sveiby model (Sveiby, 1997). A proposal for quantifying the competencies is also represented by the project management tests. The proposed CAT mechanism lies on an improved knowledge representation successfully used by the SinPers project (Bodea, 2007). The immediate benefit of this mechanism is the reduction of conception time and the increased configurability of assessment in project management. The applicability of this mechanism lies in certification processes development or self-preparation processes for project management exams, with project management being considered a strategic domain in economic activities. The chapter describes the application of Computer Adaptive Testing to the domain of project management. An application of the technique to a group of 75 students (and a further control group of 75) undertaking a public project management exam is reported, and the conclusion is that the approach is effective. The average grade of both groups was similar, but more extreme values were obtained in the second group (the one that didn’t use CAT during the preparation period). The percentage of failure was much higher in the group who prepared with classical e-testing tool.
REFERENCES
Al-A’li, M. (2007). Implementation of an improved adaptive testing theory. Journal of Educational Technology & Society, 10(4), 80–94.
Armenski, G. S., & Gusev, M. (2005). Infrastructure for e-testing. Electronics and Energetics, 18(2), 181–204.
147
A Tool for Adaptive E-Assessment of Project Management Competences
Barla, M., Bieliková, M., Bou Ezzeddinne, A., Kramár, T., Simko, M., & Vozár, O. (2010). On the impact of adaptive test question selection for learning efficiency. Computers & Education, 1–12.
Beldagli, B., & Adiguzel, T. (2010). Illustrating an ideal adaptive e-learning:A conceptual framework. Procedia Social and Behavioral Sciences, 2, 5755–5761. doi:10.1016/j.sbspro.2010.03.939
Bjorner, B. J., Chang, C.-H., Thissen, D., & Reeve, B. (2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16(supplement 1), 95–108. doi:10.1007/s11136-007-9168-6
Bodea, C. (2007). SinPers-An innovative learning system in project management. Proceedings of 21st IPMA World Congress, Cracow, Poland. ISBN 978-83-89541-93-2
Bodea, C., & Dascalu, M. (2009). A parametrized Web-based testing model for project management. Proceedings of the Advances in Web-based Learning, ICWL 2009 (pp. 68–72). Aachen, Germany: Springer.
Bodea, C. N., & Dascalu, M. (2009). Designing project management tests based on semantic nets and concept space graph. Creating Global Economies through Innovation and Knowledge Management: Theory & Practice, Proceedings of the 12th International Business Information Management Association Conference (pp. 1232 – 1236). Kuala Lumpur, Malaysia: IBIMA.
Bodea, C. N., & Dascalu, M. (2009). Testing as a learning tool in the project management competence development environments. Studia Informatica, 3, 11–14.
Casanova, D., Holmes, B., & Huet, I. (2009). Aiding academics to move from knowledge management to knowledge creation: Conceptualisation of a personal academic environment (PAE). Proceedingss MICTE 2009 (pp. 481–486). Badajoz, Spain: Research, Reflections and Innovations in Integrating ICT in Education.
Dagger, D., Wade, V. P., & Conlan, O. (2004). Developing active learning experiences for adaptive personalized e-learning. In Adaptive Hypermedia and Adaptive Web-Based Systems (pp. 55–64). Berlin / Heidelberg, Germany: Springer. doi:10.1007/978-3-540-27780-4_9
Delcea, C., & Dascalu, M. (2009). Knowledge strategies tools for managing enterprise crisis. The 4th International Conference on Knowledge Management: Projects, Systems and Technologies (pp. 115-117). Bucharest, Romania: “Carol I” National Defense University, Romania.
Desmarais, M., & Pu, X. (2005). Computer adaptive testing: Comparison of a probabilistic network approach with item response theory. Lecture Notes in Computer Science, 3538, 392–396. doi:10.1007/11527886_51
Eggen, T. J. H. M., & Straetmans, G. J. J. M. (1996). Computerized adaptive testing for classifying examinees into three categories. Retrieved from www.cito.co.uk/ research_and…/ ~/…/ cito_report96_3.ashx
Fabra, E., & Camisón, C. (2009). Direct and indirect effects of education on job satisfaction: A structural equation model for the Spanish case. Economics of Education Review, 28, 600–610. doi:10.1016/j.econedurev.2008.12.002
Fetzer, M., Dainis, A., Lambert, S., & Meade, A. (2008). Computer adaptive testing (CAT) in an employment context. Retrieved from www.previsor.com/ pdf/ WPCAT.pdf
148
A Tool for Adaptive E-Assessment of Project Management Competences
Garofano, C., & Salas, E. (2005). What influences continuous employee development decisions? Human Resource Management Review, 15, 281–304. doi:10.1016/j.hrmr.2005.10.002
Guzmán, E., Conejo, R., & Pérez-de-la-Cruz, J.-L. (2007). Adaptive testing for hierarchical student models. User Modeling and User-Adapted Interaction, 17(1-2), 119–157. doi:10.1007/s11257-006-9018-1
Hardas, M. (2006). A novel approach for test problem assessment using course ontology. Retrieved from http://www.medianet.kent.edu/ techreports/ TR2006-11-02-Thesis-ManasHardas/ final-Thesis-ManasHardas.pdf
ICB. (2008). International Project Management Association competence baseline. Retrieved from http://www.pm.org.ro/ ICB-V-3-0/ ICB-V-3-0.pdf
Kröll, M. (2007). Limits and possibilities for competence development in firms. European Conference of Educational Research (ECER). Ghent. Retrieved December 12, 2009, from http://www.vet-research.net/ ECER_2007/ sep21/ session_10c/ 985/ 985_pdf/ attach/ Kroll_ecer2007_p985.pdf
Lee, Y.-J., Palazzo, D., Warnakulasooriya, R., & Pritchard, D. (2008). Measuring student learning with item response theory. Retrieved from http://relate.mit.edu/ papers/ CpLPW08Second AttemptIRTPhys RevSTPhys EducRes3_010 102.pdf
Linacre, J. M. (2000). Computer – adaptive testing: A methodology whose time has come. Retrieved from www.rasch.org/ memo69.pdf
Lysagh, R. M., & Altschuld, J. W. (2000). Beyond initial certification: the assessment and maintenance of competency in professions. Evaluation and Program Planning, 23, 95–104. doi:10.1016/S0149-7189(99)00043-9
Marimuthu, M., Arokiasamy, L., & Ismail, M. (2009). Human capital development and its impact on firm performance: Evidence from developmental economics. The Journal of International Social Research, 2(8), 265–272.
Phobun, P., & Vicheanpanya, J. (2010). Adaptive intelligent tutoring systems for e-learning systems. Procedia Social and Behavioral Sciences, 2, 4064–4069. doi:10.1016/j.sbspro.2010.03.641
Rudner, M. (1998). An online, interactive, computer adaptive testing tutorial. Retrieved from http://edres.org/ scripts/ cat
Stergiou, N., Georgoulakis, G., Margari, N., Aninos, D., Stamataki, M., & Stergiou, E. (2009). Using a Web-based system for the continuous distance education in cytopathology. International Journal of Medical Informatics, 78(12), 827–838. doi:10.1016/j.ijmedinf.2009.08.007
Suikki, R., Tromstedt, R., & Haapasal, H. (2006). Project management competence development framework in turbulent business environment. Technovation, 26, 723–738. doi:10.1016/j.technovation.2004.11.003
Sveiby, K. E. (1997). The new organizational wealth: Managing & measuring knowledge based assets. San Francisco, CA: Berret-Koehler Publishers Inc.
Tao, Y.-H., Wu, Y.-L., & Chang, H.-Y. (2008). A practical computer adaptive testing model for small-scale scenarios. Journal of Educational Technology & Society, 11(3), 259–274.
van der Linden, W., & Glas, C. A. (2010). Elements of adaptive testing. London, UK: Springer Science+Business Media.
Waters, R., & McCracken, M. (1997). Assessment and evaluation in problem-based learning. Retrieved from http://www.succeed.ufl.edu/ papers/ fie97/ fie97-010.pdf
149
A Tool for Adaptive E-Assessment of Project Management Competences
ADDITIONAL READING
Baylari, A., & Montazer, G. (2009). Design a personalized e-learning system based on item response theory and artificial neural network approach. Expert Systems with Applications, 36, 8013–8021. doi:10.1016/j.eswa.2008.10.080
Carceles-Poveda, E., & Giannitsarou, C. (2007). Adaptive learning in practice. Journal of Economic Dynamics & Control, 31(8), 2659–2697. doi:10.1016/j.jedc.2006.09.004
Chen, T. Y., Kuo, F.-C., Merkel, R. G., & Tse, T. (2010). Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software, 83(1), 60–66. doi:10.1016/j.jss.2009.02.022
Cheng, S.-C., Lin, Y.-T., & Huang, Y.-M. (2009). Dynamic question generation system for web-based testing using particle swarm optimization. Expert Systems with Applications, 36(1), 616–624. doi:10.1016/j.eswa.2007.09.064
Forbey, J. D., & Ben-Porath, Y. S. (2007). Computerized Adaptive Personality Testing: A Review and Illustration With the MMPI–2 Computerized Adaptive Version. Psychological Assessment, 19(1), 14–24. doi:10.1037/1040-3590.19.1.14
Frey, A., & Seitz, N.-N. (2009). Multidimensional adaptive testing in educational and psychological measurement: Current state and future challenges. Studies in Educational Evaluation, 35(2-3), 84–94. doi:10.1016/j.stueduc.2009.10.007
Haley, S. M., Ni, P., Hambleton, R. K., Slavin, M. D., & Jette, A. M. (2006). Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank. Journal of Clinical Epidemiology, 59, 1174–1182. doi:10.1016/j.jclinepi.2006.02.010
Ivan Chang, Y.-c., & Ying, Z. (2004). Sequential estimation in variable length computerized adaptive testing. ournal of Statistical Planning and Inference, Vol. 121, No. 2, 249-264.
Lilley, M., Barker, T., & Britton, C. (2004). The development and evaluation of a software prototype for computer-adaptive testing. Computers & Education, 109–123. doi:10.1016/j.compedu.2003.12.008
Lunz, M. E., Bergstrom, B. A., & Gershon, R. C. (1994). Computer adaptive testing. International Journal of Educational Research, 21(6), 623–634. doi:10.1016/0883-0355(94)90015-9
Preux, P., Delepoulle, S., & Darcheville, J.-C. (2004). A generic architecture for adaptive agents based on reinforcement learning. Information Sciences, 161(1-2), 37–55. doi:10.1016/j.ins.2003.03.005
Ruiz, B., Fitz, P. A., Lewis, C., & Reidy, C. (1995). Computer-Adaptive Testing: A New Breed of Assessment. ournal of the American Dietetic Association, Vol. 95, No. 11, 1326-1327.
Smoline, D. V. (2008). Some problems of computer-aided testing and ‘‘interview-like tests’’. Computers & Education, 51, 743–756. doi:10.1016/j.compedu.2007.07.008
Stricker, L. J., Wilder, G. Z., & Rock, D. A. (2004). Attitudes about the computer-based test of English as a Foreign Language. Computers in Human Behavior, 20(1), 37–54. doi:10.1016/S0747-5632(03)00046-3
Triantafillou, E., Georgiadou, E., & Economides, A. A. (2008). The design and evaluation of a computerized adaptive test on mobile devices. Computers & Education, 50(4), 1319–1330. doi:10.1016/j.compedu.2006.12.005
Walker, J., Böhnke, J. R., Cerny, T., & Strasser, F. (2010). Development of symptom assessments utilising item response theory and computer-adaptive testing—A practical method based on a systematic review. Critical Reviews in Oncology/Hematology, 47–67. doi:10.1016/j.critrevonc.2009.03.007
150
A Tool for Adaptive E-Assessment of Project Management Competences
Wang, T.-I., Wang, K.-T., & Huang, Y.-M. (2008). Using a style-based ant colony system for adaptive learning. Expert Systems with Applications, 34, 2449–2464. doi:10.1016/j.eswa.2007.04.014
Weiss, D. J. (1985). Adaptive Testing by Computer. Journal of Consulting and Clinical Psychology, 53(6), 774–789. doi:10.1037/0022-006X.53.6.774
Wiener, A., Marcus, E., & Mizrahi, J. (2007). Objective measurement of knee extension force based on computer adaptive testing. Journal of Electromyography and Kinesiology, 17, 41–48. doi:10.1016/j.jelekin.2005.12.004
Young, R., Shermis, M. D., Brutten, S., & Perkins, K. (1996). From Conventional to Computer-Adaptive Testing of ESL Reading Comprehension. System, 24(1), 23–40. doi:10.1016/0346-251X(95)00051-K
KEY TERMS AND DEFINITIONS
Competence: A specific, identifiable and measurable knowledge, skill, ability and/or other deployment-related characteristic (e.g. attitude, behaviour, physical ability) which a human resource may possess and which is necessary for, or material to, the performance of an activity within a specific business context.
Computer Adaptive Testing: An algorithm used in computer based testing in which the difficulty level of the next question is established according to the correctness of the answer given at the current question.
Concept Space Graphs: Directed graphs, in which each node and each edge has weights and path weights can be calculated according to them; nodes represent concepts and edges represent the relationship between concepts (hierarchy, logical constrains).
Educational Ontology: A formal representation of the knowledge by a set of concepts and the relationships between them within a domain, used for educational purposes. Item Response Theory: A paradigm in psychometrics used for the design, analysis and scoring of tests or questionnaires, measuring abilities, attitudes, or knowledge.
Item Pool: An amount of questions, from each one has to choose, at each step, when using Computer Adaptive Testing principle.
Semantic Network: A form of knowledge representation, evoking a graph, which consists of edges and vertices; the vertices represent concepts.

Leave a Reply

Your email address will not be published. Required fields are marked *