THE BEHAVIORAL SIDE OF RECOMMENDATION AGENTS: A BIBLIOMETRIC REVIEW O LADO COMPORTAMENTAL DOS AGENTES DE RECOMENDAÇÃO: UMA REVISÃO BIBLIOMÉTRICA

Recommendation agents have been used to assist consumers in online purchase for almost 20 years. Their use has been studied in academic research with two different approaches. The first one addresses computational problems related to generating accurate recommendations. The other seeks to understand how user interaction with recommendation agents can alter behaviors in online shopping. Through bibliometric and scientometric methods, this study looked for the most influential papers, authors and journals in the field of behavioral recommendation research. In the present work, only articles investigating behavioral aspects of recommendation usage were considered. The identified articles were analyzed in terms of their methodology, variables and repercussion. At the end, a total of 175 articles published in journals from many different fields of academic research were found, attesting the multidisciplinary nature of this topic. Most of the studies were empirical investigations using experimental methodology, however theoretical papers showed to be more influential. It was possible to identify 29 different dependent variables used to measure the effects of recommendations in online assisted purchase. The 19 independent variables used in these studies were related to characteristics of the recommendation agent, user characteristics or vendor characteristics. Results also showed that the field still lacks confirmatory studies capable of creating a greater assurance for the knowledge already developed in the field.


Introduction
Recommendation agents (RA) are decision aids designed to identify consumer preferences based on previously collected data, in order to present personalized information about the products that better fit their needs (Bodapati, 2008;Shih et al., 2002;Wang & Benbasat, 2007). Kardan and Ebrahimi (2013) define them as personalized decision support systems capable of predicting the utility of an item for a particular user and suggesting it to her (Kardan & Ebrahimi, 2013). Considering they are personalized systems, recommendation agents can help consumers to make purchase decisions at a certain point in time by giving them advices tailored specifically to their needs (Shani & Gunawardana, 2011).
Once they give advices based on previously specified user preferences, recommendation agents have the potential to reduce information overload and search complexity but at the same time improve decision accuracy (Shani & Gunawardana, 2011;Xiao & Benbasat, 2007). Recommendation agents, therefore, can influence not only the way users make decisions while searching for product alternatives, but also which, among all available options, they will consider. In particular, "recommendations cause consumers to rely less on the utility difference between a newly inspected product and the best previously encountered one, to make broader comparisons among the set of inspected products" (Dellaert & Häubl, 2012, p. 285).
Given their constitution and purposes, scientific research on recommendation agents is two-folded. One field of investigation seeks to create more efficient algorithms destined to a better understanding user preferences based on available data. The other field is concerned with behavioral aspects of recommendations, looking for a deeper comprehension of how interactions with recommendation agents can influence the behavioral aspects of Internet usage. Although both fields of research are still relatively recent, there is already a great bunch of research published in both sides, with a greater emphasis on the computational issues. Kim and Chen (2015) have already performed a scientometric review of the term recommendation systems and its derivatives, but their focus was broader, which consequently led them to focusing solely on the algorithmic features of recommendation agents, leaving aside behavioral aspects of recommendation agents. This is understandable, given the fact that in the present review it was possible to identify a total of 979 articles using the words recommendation systems or recommendation agents, but only 175 of them where outside of the computational domain. The disparity between the amount of scientific production using algorithmic or modeling approaches and other types of studies, together with the fact that there is still a need for understanding and improving users responses to this mechanisms, calls for a thorough review of what has been published so far and what are the gaps that remain to be filled.
Using a bibliometric and scientometric review of current research on users' responses to recommendations, the main goal of this paper was to analyze what has been discovered so far as well as to identify research gaps that remain to be filled after more than 20 years of investigation. For that, a thorough search was performed both in Google Scholar and Web of Science looking for all published papers using one of the following key words (or their variations): (i) recommendation agents; (ii) recommendation systems; (iii) recommender agents; and (iv) personalization agents. From the identified work, only articles using a behavioral approach were used. These articles were analyzed from a scientometric perspective, to identify quantitative aspects of scientific production in the field, such as main publishers, authors and citations. After that, the analysis was expanded to the content of the identified articles to unveil the main aspects that constitute the research tradition in behavioral recommendation agents research.
One of the main contributions presented here is to shed light over an important subject in business research, once the steady growth of ecommerce transactions has demanded each time more knowledge development to help companies to adapt their strategies to virtual environments. Besides that, it also unveils the methodologies and variables that have been used to study the subject as well as the main publishers and publications, what can be used as a shortcut for new researchers starting to develop studies about it. A third contribution is to find some important gaps that remain to be filled in order to advance in the comprehension of the consequences of recommendation use on consumer behavior. This is why at the end of the paper, a set of research propositions is being presented to suggest the possible continuation paths that the development of knowledge in this area may take.

Recommendation agents: a definition
While making a purchase, consumers tend to be attracted to more choice, but then experience difficulty due to the negative psychological consequences associated with choosing from such a large set (Schwartz, 2004). Since consumers try to reduce the cognitive effort spent to make a decision (Aljkhadar, Senecal & Daoust, 2012;Bettman, Luce & Payne, 1989;Tversky & Shafir, 1992), it is possible to find evidences to support the existence of negative relations between effort and satisfaction with the purchase (Oliver & Swan, 1989).
However, although this is true for individual efforts, consumers generally appreciate other party's efforts trying to help them (Bechwati & Xia, 2003;Mohr & Bitner, 1995). In this sense, recommendation agents can affect consumer's decision-making processes by altering the amount of effort demanded in order to make a purchase decision (Xiao & Benbasat, 2007) A typical interaction proceeds as follows: first, user's preferences are elicited (whether directly or indirectly). Based on the collected preference data, the system tries to predict how much the user would appreciate each of the available items in the catalog. Finally, those items that have the highest predicted value are presented to the user. In some recommendation agents, this terminates the interaction, in others, users continue to indicate their preferences and receive recommendations continually. The design of a recommendation agent, then, needs to consider three main factors: (i) inputs, which are related to the way user preferences are elicited; (ii) process, when recommendations are generated; and (iii) outputs, that is, the way recommendations are presented to the user (Ricci, Rokach & Shapira, 2011).
Recommendation agents, therefore, can be classified according each one of the mentioned characteristics (input, process and output) and also according to the type of algorithm used to generate the recommendations (RA type). Table 1   Navigation and layout Navigational path to product information and layout of the product information Source: elaborated by the author based on Benbasat (2007, 2014) and Yoo and Gretzel (2012) A deeper explanation of the features exposed in Table 1 can be found in Benbasat (2007, 2014) and Yoo and Gretzel (2011). As the purpose of this article is not to address technical issues but to study behavioral responses to recommendations in online environments, the focus of the following subchapters will be on the theoretical perspectives currently used to understand them.

Consumer responses to Recommendation Agents
In the face of the aforementioned definitions, it still remains unclear what is the main purpose of an online recommendation agent. Is it designed to remediate the lack of social interaction and the absence of personal sales consultation in online environments (i.e. Holzwarth, Janiszewski & Neumann, 2006)? Is it intended to help consumers to optimize their decision-making process by reducing decision effort and increasing decision accuracy  Bodapati, 2008;Fitzsimons & Lehmann, 2004;Wang & Benbasat, 2007;Zhang & Pu, 2006)? Or could it be just a new strategy companies use to persuade consumers and increase sales revenues?
Independently of the goal that motivated the inclusion of decision aids at a website, it seems reasonable to suggest that the ultimate measure of the RA effects from a user's perspective is advice acceptance. Accepting a recommendation means that consumers analyzed the alternative proposed by the Recommendation Agent and considered it as the best option among all available. This will happen in the case that the recommendation presented is in accordance with the personal preferences of an individual (Komiak & Benbasat, 2006;Sinha & Swearingen, 2001) or if she believes the RA is operating in her best interests (Häubl & Murray, 2006;Wang & Benbasat, 2009). Usually, online acceptance is a measure of accuracy and it can be calculated by different methods such as: (i) selection of nondominated alternatives, (ii) utility values, (iii) selection of target choice and (iv) selection of target choice among k-best items (Zhang & Pu, 2006).
Other important measure of RA influence, when one assumes it is designed to help consumers to optimize their decision-making process, is cognitive effort. Wang (2005) argues that there is an important role played by consumer's cognitive effort in their evaluations and acceptance of the recommendation agents. It is also argued that consumers tend to focus more in reducing effort than in increasing decision accuracy because feedback on effort expenditure can be accessed immediately while feedback on accuracy is subject to both delay and ambiguity (Todd & Benbasat, 1992;Wang, 2005). In line with that, thus, if two strategies will produce the same level of accuracy, the one which is expected to require less effort will be preferred (Todd & Benbasat, 1994).
Cognitive effort is frequently measured in two ways: (i) consideration set size and (ii) decision time (Wang, 2005). A consideration set is the amount of options a consumer considers seriously before decision-making (Häubl & Trifts, 2000). Consequently, too many options included in a consideration set will demand higher cognitive effort than smaller sets.
Recommendation Agents can actually decrease set size when consumers find them trustworthy (Häubl & Murray, 2006;Häubl & Trifts, 2000). Other measure for cognitive effort is decision time, which can be computed directly by the time consumers spend in making a decision. Some authors have also argued for the use of indirect measures for cognitive effort, such as perceived cognitive effort (Kleijnen, De Ruyter & Wetzels, 2007;Kurzban et al., 2013). They argue user's perception of cognitive effort can be more determinant to intention and future behavior because it deals with the impressions primed in consumer's memory, especially because a consumer will rarely monitor the exact time spent to make a decision.
Intention to use has also been considered an important measure to RA effects. Several studies have demonstrated that effort and quality are two important variables influencing users' choice behavior and their intentions to use decision aids (e.g., Payne, 1982). Dabhlokar and Bagozzi (2002) propose a model to measure intention to use an online system based on the reported probability of using it in the future. Wang and Benbasat (2009) also developed a similar scale adapted from Davis (1989) to be used specifically in decision aids.
Satisfaction is also considered to be an important driver of future behavior and an It is also important to consider that one important response to recommendations can arise in some cases. According to Fitzsimons and Lehmann (2004), although much of the literature suggests that opinions and recommendations are desirable in decision-making, this only happens when the recommendation is consistent with individual choice preferences.
Consequently, when recommendations contradict the consumer's initial impressions of choice options, there will be an increased level of difficulty in making the decision and, at the same time, an individual tendency to choose the alternative rejected by the recommender (Fitzsimons & Lehmann, 2004).
This kind of response can happen when the individual feels that, rather than a mechanism for facilitating decision-making, the recommendation agent is purposely limiting the consideration set, restricting her freedom of choice. According Fitzsimons and Lehmann (2004), based on the theory developed by Brehm in 1960, threats to freedom can motivate an individual to adopt behaviors that seek to regain the freedom once threatened or lost, even if these behaviors are not congruent with their immediate needs. The motivation for the recovery of this freedom is called psychological reactance. Fitzsimons and Lehmann (2004) believe that reactant behavior can be stimulated when the recommendations are unwanted. They found that when the recommendation is contrary to personal choice preferences, some unexpected patterns emerge. As decision-making difficulty increases, given the conflicting information, choice and confidence in the non-recommended alternative significantly increase, giving room for a reactant behavior.  Lee and Lee (2009)

reached convergent conclusions conducting an experimental study
at an e-commerce store. The empirical results of their work have shown that user expectations for personalized service induces the perception of usefulness, because choosing among too many alternatives may be a nuisance to the decision maker. Wang and Benbasat (2009) investigated the impact of perceived restrictiveness on user behavior and found that it significantly affects the perceived cognitive effort, advice quality and consumer's intentions to use online decision aids. They also found that decision strategy plays a significant role in perceived restrictiveness, in that "the additive-compensatory aid is perceived to be less restrictive, of higher quality, and less effortful than the elimination aid, whereas the hybrid aid is not perceived to be any different from the additive-compensatory aid" (Wang & Benbasat, 2009, p. 293).

Methodology
This review aims to investigate the intellectual landscape in recommendation agents research, looking for identifying thematic patterns, landmark articles and emerging trends. For that, it uses mathematical and statistical tools to analyze and quantify published documents. It is, therefore, intended to collect and classify, in a systematic way, the totality of data related to the production, conservation, circulation and utilization of every type of published material (Pritchard, 1969) related to recommendations, with a specific focus on behavioral research.
It is important to note that the boundaries of this subject are still unclear because it deals with interdisciplinary and relatively new phenomena (Kim & Chen, 2015). In order to give a clear delimitation for the purposes of this article, studies in the field were classified in two different categories, one destined to solve computational problems and the other addressing behavioral and managerial problems related to the use of recommendations. As already mentioned, the present study focuses in the later.
Since the classification in the proposed categories is arbitrary and is not still defined in previous published papers, the first step of this research was to find all published papers with either one of the mentioned approaches. The data collection was made in two different sources: Google Scholar and Web of Science Database. That was necessary because the intention in this exploratory phase was to find every published paper on the subject. For that, four key words and their variations were used: (i) recommendation agents; (ii) recommendation systems; (iii) recommender agents; and (iv) personalization agents in both databases. These two datasets of bibliographic records were retrieved using a topic search and The data collection process resulted in a total of 1886 articles that contained either the expression recommendation agent, recommender system, recommendation system or personalization agent. From these first results, duplicated articles were eliminated, as well as the articles which did not relate to the field of recommendation agents (i.e. articles that used the word recommendation and its variances but referred to other subject matters). Articles written in languages other than English were excluded too. Dissertation theses were also  The citation analysis aimed at identifying the most influential papers and publishers and also the most cited authors. For that, the number of citations, as found in the mentioned databases, was used as a measure of their impact and influence over research in the field. The content analysis performed on the selected papers intended to identify the main theories and methodologies used in the studies.

Results
As already mentioned in the methodology session, a total of 979 articles were found that used the key words established for Phase 1. These articles were classified by their focus as algorithmic if they studied computational factors of recommendations or as behavioral if they studied user's responses to recommendations. Figure 2 shows a comparison of the number of published articles by year in each one of these two groups. Other group within the methodological classification of work identified in this research was related to econometric models. These models have a very specific goal in recommendation agents research, which is to understand and predict the impacts of recommendation agents adoption by firms in terms of revenues and other measures of a company's performance. The other articles did not show the same influence in terms of citations, although they corresponded to a quarter of the total articles sampled. Table 2 shows a summary of the results. The next step of the results analysis was to identify the most influent publications, researchers and publishers in the field of recommendation agents. From the 40 most cited articles, 42.5% were empirical studies using experimental methodology. Another 35.0% were theoretical investigations based on literature reviews.

Revista Brasileira de Gestão e Inovação -Brazilian Journal of Management & Innovation
v.5, n.  Looking solely to the number of citations, it is possible to infer that theoretical articles were more influential, accounting for 52.78% of the total citations. From these, the two most cited articles accounted for 29.9% of the citations. Articles using experimental methodology corresponded to 35.55% of the total citations. These results help to reinforce the idea that theoretical works play a significant role in the production of knowledge. Table 3 depicts a summary of the findings. more studies on the subject (Guedes & Borschiver, 2005). Subsequently, more journals show interest on the topic until some of them arise as the more productive in a certain field.
Looking at the media where the identified articles were published, it is possible to understand the multidisciplinary aspects of behavioral recommendation agents research. Articles presenting research results on recommendation agents could be found in publications destined to psychology, management, computer sciences and marketing. It is also possible to note the importance of international events for generating new knowledge in the field, given the number of citations generated from articles in these kinds of publications. Table 4 shows an outline of such results. article published and whose citations exceeded a certain amount (300 in the case) are shown.
Only the mentioned authors are being outlined because the number of articles published indicates a continuation and consistency in trying to solve research problems in this subject matter. The number of citations was also important because it indicates the influence their studies had in subsequent research. Results showed that only a few number of researchers has actually had a consistent number of publications with a significant reach. Only eight authors had more than three papers published with high repercussion. Professor Izak Benbasat, although was not the first author in most of the papers, showed to be the most prolific author in behavioral recommendation research when it comes to the number of publications. This suggests that he may be the main author encouraging the continuation of research in this matter. Besides him and a few others, the rest of the authors appear to have demonstrated only a situational interest in the field, but it was not maintained over time.

Theories in behavioral ra research
Although smaller in number, theoretical articles showed to have a greater impact in the field, if one looks at the number of citations. It is also important to note that the identified studies were not actually interested in developing a new theory, they rely in other theoretical background to derive explanations for the effects of using recommendation agents. The theoretical articles already mentioned generally resume to structured literature review or try to propose models based on previous theory from other fields.
The six theoretical perspectives regularly used by researchers in order to better comprehend the effects of recommendation agents on consumer behavior were: (i) theories of and (iv) theory of social response. The theories of human information processing have been mainly used in investigations studying RA-assisted consumer decision-making. The following three were used to study user's evaluations of RAs and their adoption intention. Finally, the theory of interpersonal similarity has been used by both streams of research. Other theoretical perspective is the theory of social response which defends that people relate to computational agents the same way they do with people. Table 6 presents some assumptions derived form the six theoretical perspectives mentioned in this brief review. Derived assumption Related variables

Human information processing
Consumers will appreciate decision aids trying to reduce cognitive effort spent to make a decision.
-Decision time -Perceived cognitive effort

Theories of satisfaction
User satisfaction is an important measure of future intention to use the system.
-Satisfaction with the system -Satisfaction with search process -Satisfaction with decision

Theories of trust formation
Consumers will only accept recommendations from sources they trust.
-Recommendation acceptance -Trust in RA (trasparency, competence and confidence) -Reactance The technology acceptance model Intention to adopt a new technology is determined by the perceived usefulness of using the technology and the perceived ease of use of the technology.
-Recommendation acceptance -Adoption intention Theory of interpersonal similarity Consumers will be more prone to accept recommendations when they perceive similarities between them and the RA.
-Recommendation acceptance -Perceived product fit Theory of social response RAs will be more persuasive when they leverage social aspects from their users.
-Recommendation acceptance -Social presence Source: The authors

Experimental studies in behavioral ra research
At the end, an analysis of the quantitative studies was performed in order to identify the variables they used and how such variables were linked to the use of recommendation agents. Based on these results, the studied variables were classified into five categories: (i) direct behavior: variables that measured certain acts of the user not based on self reports; (ii) perception: measures of the impressions reported from subjects after interactions with RA's; (iii) evaluation: measures assessing the degree to which consumers reported to have been impacted by the use of recommendations; (iv) intention: planned behavior caused by the use of recommendations; and (v) attitude. The most used variables in the studies were intention to use a RA (19 studies), trust (12 studies), time to make a decision (6 studies), user satisfaction (9 studies) and purchase intention (9 studies). A summary of the data found is presented in Table 7. Perceived transparency 2 * Classification: (1) direct behavior, (2) perception, (3) evaluation, (4) intention, (5) attitude. Source: The authors As independent variables, empirical studies generally considered factors related to RA characteristics, user characteristics and vendors characteristics. Figure    It is interesting to note that some variables were used both as dependent and independent variables in some studies. It happened more specifically with variables related to trust and attitudes. Although it seems paradoxical at a first glance, this could indicate that such variables may be operating as mediators or moderators of the recommendation effects over consumer behavior. Those relations have actually been hypothesized by Benbasat (2007, 2014), but they still have not been tested empirically.
Finally, experimental studies were analyzed to quantify the type of products they used in their designs. The products related to a wide range of consumption experiences going from tangible products to highly intangible services as intention to adopt a future behavior. A great deal of them, however, focused on tangible products. Table 8 presents the findings. solution, in behavioral recommendation research, the amount of publication did not reduce in the same proportion.
After a detailed search for already published papers using recommendation agents (and its variants) as key words, a total of 175 articles were found that could be classified as studying behavioral aspects of recommendation agents use. From these, the most influential articles in number of citations were structured literature reviews with theoretical propositions.
It was possible to conclude that the amount of relevant researchers in the field (as measured by the impact of their work in terms of citation) is reduced and the number of authors with more than three relevant published articles is even smaller.
As for the dependent measures used to identify the impacts of recommendation agents on consumer behavior, 29 different measures were found and classified into five categories: (i) direct behavior; (ii) perception; (iii) evaluation; (iv) intention; and (v) attitude. The most used measures were decision effort, trust, intention to use the RA and user satisfaction.
Surprisingly, recommendation acceptance was only used as a dependent measure in two studies. The independent measures found were related to RA characteristics, user characteristics and vendor characteristics.
It is also interesting to note that no replications of any of the mentioned studies were found. The absence of replications in this case may constitute a problem for consolidation of the knowledge in the field. Even when using the same variables, studies tended to manipulate experimental sets in such a way that would turn thorough comparisons inaccurate. The complexity of the subject and the amount of new problems that arise from such a prominent field indicates that confirmation studies should be performed in order to reach a better understanding of the effects of assisted purchase. Future studies could focus on replications and also confirmation of the already existent theoretical models to lead knowledge in the field to further steps.

Challenges for behavioral recommendation research
Research studying the effects of recommendations on consumer behavior is still far from achieving a thorough comprehensive model. Current theoretical models destined to such challenge continue to lack empirical proof. Still, some central questions remain to be answered and others continue to emerge as new ecommerce stores start to include recommendations in their web pages.
The main gap found is related to the determinants of recommendation acceptance. Although it could be considered the major metric of recommendation agents' effectiveness, only two studies used it as a dependent variable. Even such studies failed to understand the main drivers of recommendation acceptance (i.e. Adomavicius et al., 2013;Gershoff et al., 2003). These studies have discovered that finding the best alternative according to inferred users' characteristics is only a small part of a much more complex phenomenon (Adomavicius et al., 2013).
System accuracy, thus, is not the only driver of consumer's responses to recommendations (Cooke et al., 2002;McNee et al., 2006). Some new metrics need to be considered for evaluating the effects of recommendations. Rana and Jain (2012) have already outlined the role that contextual factors play in consumers' decision processes. They present a need for the development of recommendation agents that could handle the temporal dynamics of the user needs as well as system content and, accordingly, present modified recommendations to the users in real-time (Rana & Jain, 2012).
Considering the aforementioned, it is possible to suppose that consumers are especially sensitive to the recommendation context at an early stage, but as experience with the website grows, so does the level of trust that the consumer places in the recommendations (Cooke et al., 2002). That is, the dissociation between agent and item evaluations disappears with repeated visits to a specific website. Recommendation agents, then, should consider previous user experience in order to incrementally adapt and simplify the way preferences are elicited and recommendations are presented to a specific user's reality.
As the present study showed, research in the field has ignored those factors. A great amount of studies so far has been concerned with analyzing responses to recommendations generated by explicit elicitation methods in single interactions. This focus disregards current trends in the way recommendations have been applied in websites. Various types of companies providing products or services on the Internet (i.e. Amazon, Aliexpress, Saraiva, Netflix, Facebook, Youtube) have preferred to apply implicit elicitation methods to generate recommendations.
Customers of these companies have also been characterized for repeated visits to their web pages. Empirical investigation, with some rare exceptions (see Xiao & Benbasat, 2003), has also failed to consider the effects of assisted purchase across time. So, future research should also consider the effects of several interactions with recommendations from the same website to reach a better understanding of consumers' responses to recommendations.
Finally, there is an important issue that all mentioned studies did not consider, related to consumers' attitudes towards recommendations. It is logical to suppose that one factor influencing recommendation acceptance and decision effort would be the projected cost of a wrong decision. For example, buying the wrong laptop seems to be more costly than buying a bad wine or listening to some unwanted music. In this sense, it appears to be true that consumers would be more prone to accept recommendations when the projected costs of making the wrong choice is smaller. No study was concerned with this issue so far. Figure 3 shows a final summary of proposed research that could fill the gaps identified in the present paper.

Product characteristics
Identify how the importance attributed to the product and the projected loss of a wrong decision may hinder recommendation acceptance Source: The authors