臺大管理論叢 NTU Management Review VOL.28 NO.3

Use of Implicit User Feedback to Support Semantics-Based Personalized Document Recommendation 94 ontology to facilitate concept-level document representation, and 528 papers served as the pool from which recommendations would be made. To control the influence of website design, the same user interface was used for each treatment. We employed a mixed design experiment to evaluate the performance of the ICE technique, including one between factor (recommendation technique) and one repeated measure factor (recommendation stage). The experiment has four treatments: ICE and the three benchmark comparison techniques (ECE, keyword-based, and random-based). While ECE and ICE employ the same document recommendation approach, ECE gathers users’ interests via explicit ratings provided by the users. Hence, if the input information differs between ICE and ECE, the recommendation results are also different, even though the processing is the same. The keyword-based technique recommends documents to users by comparing keywords pertaining to the users’ three favorite documents with keywords pertaining to the documents that have not yet been recommended (Liang et al., 2008). The higher the number of matching keywords that pertain to an unread document, the higher the interest score assigned to that document. The random-based technique recommends documents via random sampling of the documents that have not yet been recommended. Hence, the random-based technique is not a personalized recommendation system. In order to enhance the recommendation quality of the experimental recommendation systems, ten documents are recommended to each subject at each different stage during the experiment. The experimental procedures are explained in detail in Section 4.2. 4.1 Measurements of System Performance This study measures system performance based on the user’s preference for the recommended documents and the user’s perception of the quality of the recommendations. Recommendation quality was measured using three items adapted from the literature (Doll and Torkzadeh, 1988) and one item developed by our study. System interface quality was carefully controlled, since any variation might affect users’ perception of the recommendation quality. The perceived quality of the system interface was also measured to determine whether users’ perceptions of the UI quality were consistent among the four treatments. Hence, we adopted three items related to system interface quality from a previous study (Doll and Torkzadeh, 1988) and developed one new item to measure system interface quality. All items were measured using Likert scales ranging from 1 (strongly disagree) to 5 (strongly agree). All items are shown in Table 3.