We’ve had some trouble recently with posts from aggregator links like Google Amp, MSN, and Yahoo.
We’re now requiring links go to the OG source, and not a conduit.
In an example like this, it can give the wrong attribution to the MBFC bot, and can give a more or less reliable rating than the original source, but it also makes it harder to run down duplicates.
So anything not linked to the original source, but is stuck on Google Amp, MSN, Yahoo, etc. will be removed.
How much are you paying for the MBFC API? The page says it isn’t free. I’ll give you an API endpoint which will check sources against https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources, if you pay me half of whatever you were paying MBFC previously. That list is quite a lot better than relying on MBFC.
I already scraped the list. It’ll take around an hour for my script to finish going down the sources and assigning web sites to each one, but I can have a working API endpoint for you tomorrow morning. I can do the bot part also, if you prefer. That’s probably easier than making a new endpoint and hooking it to a bot and debugging the connection and all.
Like I said, I think the idea that readers won’t be able to determine that Breitbart is unreliable is missing a pretty big elephant in the misinformational room. If the issue that’s causing you to keep MBFC is finding a better source that’s programmatic, though, then solving that is almost trivially easy and at least seems like some kind of step forward.
MBFC API is free as they gave us access for us as a Non Profit.
We already had in mind adding these sources to our bot but we didnt had the time and knowledge how to scrape that. Personally i would like to host it on our own server so that we dont require you to use your own money just for one bot, in what programming language did you write it?
Thanks a lot!
Rooki
Here you go:
https://ponder.cat/wp/wp-sources.zip
It’s in python, suitable for sticking directly into the bot if the bot is in python. There are docs. It’s a first cut. How did you envision this working? I can make a real API, if for some reason that makes things easier, but it’s not immediately obvious how it would get integrated into things.
Running it on the last 50 articles posted to /c/politics, we see:
It’s more complex to use this than MBFC, because there’s a lot more depth to the rankings, and sometimes human judgement is needed to assign scores. There’s a category “needinfo,” meaning it’s necessary to know what topic is being discussed or when an article was written, because of an ownership change or similar factor. I’ve applied that judgement above. That, to me, is a good thing. It means the bot is grounded in something, and not just blithely spitting out arbitrary scores without bothering to ground them in any reality.
In practice, I think it would be realistic to assign a single reliability ranking to most of the “needinfo” sources. You can manually edit the .json data to do so. Almost all of the posts are going to fit into one of Wikipedia’s categorizations or another. Newsweek is unreliable, The Guardian is reliable, and so on.
I think most of the mixed-consensus sources can be used without a second thought. Mostly, the questions about them boil down to open partisanship of the source, which for a political community is perfectly fine as long as they’re trustable factually.
If you want me to boil this down further, so that it gives a single “yes” or “no” score to each source, I can do that and probably keep almost all of the accuracy of the rankings, now that I’ve looked at it for a little while.
When you talk about “adding” this to the bot, are you proposing to still have MBFC be the main source, with this as a footnote? A lot of the criticism of the bot is on the grounds that MBFC is a very bad source for judging reliability, so I would question the idea of keeping it on as the primary source.
Nice work, thanks for contributing!
By “adding” i mean adding it into the field higher than MBFC ( as i personally think wikipedia is a little bit better for that ).
new:
I would like to implement your code into the bot myself so i can learn how you would do it. If you are willing to share your code, please send me a github link ( or invite me if you want it to be private between you and me ) or if its super simple just send it in the dms.
I already sent it. It’s here:
https://ponder.cat/wp/wp-sources.zip
Edit: You don’t need to do the import initially, since there’s already a sources file with some small modifications. The import is the only complicated part. Use categorize.py to categorize a source, or lookup.py to run a quick command-line test.
Ok i will look into it, thanks i thought it was just the sources not the code.
Ok i implemented it into the bot and it took about 1 hour and 6 minutes to fetch all links and i am now implementing the part where it is inserted into the new text.
Sounds good. If you redid the import, I think you’ll want to make some manual fixes to the .json. Off the top of my head, I think you just need to add bbc.co.uk and aljazeera.com to the URL lists for those sources.
On a different topic: It sounds like jordanlund is saying that if he tried to remove the MBFC bot from the politics sub, he might be removed as a moderator, and replaced with someone else, and the bot would come back.
https://lemmy.world/comment/12825768
Is that true? Is the admin team mandating the use of this bot, and if so, why?
No, i dont get it from where he would get that idea, because see c/politics mods wanted the bot gone and we removed it no question asked.
@jordanlund@lemmy.world if you really dont want the bot here we can remove the bot and shut the bot down ( please consult other c/world mods too )
You mean news? Cause it’s still running on politics
Since it’s a MediaWiki page you can get Markdown source of the page with appending
action=rawquery to the URL.To be honest, that’s Rooki’s deal, but I’ll link them to this comment!
I’ll send them a link and an example of how to use it tomorrow.