News source evaluations based on fact-checking can help curb the consumption and spread of fake news on social media. Prior research has primarily considered source evaluations with intuitive icons that indicate whether or not news sources are reputable. But can we increase the power of these icons by adding more detailed information about the evaluation that explains the reasons for the icon? What additional benefit would such evaluation details bring? Would they have the same effect for both positive and negative evaluations? We conducted two online experiments to understand the effects of a source evaluation icon (a positive or negative summary of the evaluation) and more detailed evaluation information explaining the reasons for the icon. Our results show an asymmetric effect of positive and negative icons and details. Negative icons reduced the believability of the articles, but adding evaluation details supporting the icon had no additional effect. In contrast, positive icons had no significant effects, but adding evaluation details significantly increased believability. We also found that users were more likely to view the evaluation details when the content of the article aligned with their pre-existing opinions, but the valence of the icon (positive or negative) did not affect this decision. • News source evaluations based on fact-checking can help curb fake news. • Our online experiments tested the effects of source evaluation icons and details. • Negative icons reduced article believability; the details had no added effect. • Positive icons had no impact, but the added details increased believability. • Users viewed the details more for agreeable articles; icon valence had no effect.