Integrate Wikidata

[tfmorris]

There's a ton of stuff that we could be leveraging Wikidata for in addition to just linking author records to Wikidata.

• Add Wikidata IDs to works (and add OL work IDs to Wikidata)
  • Addressed in Add UI for adding work IDs #3430
• Import book series from Wikidata like A Song of Fire and Ice
• Import from Wikidata awards like NPR Top 100 Science Fiction and Fantasy Books (Q20899118)
• Import more photos, information about awards received, links to interesting places (Wikipedia in N languages)
• Import places associated with authors, works, etc to offer interesting OL stuff about places

Etc, etc. Basically use Wikidata to bling it up without having to spend a lot of time/effort.

[nichtich]

First step would be to link corresponding Wikidata ID to works, editions, authors and subjects. Links can be added in Wikidata with two Wikidata properties:

• https://www.wikidata.org/wiki/Property:P648
• https://www.wikidata.org/wiki/Property:P3847

To avoid synchronization headache it may make more sense to keep Wikidata as master for these links and harvest them regularly (plus live via SPARQL or MediaWiki API). Nevertheless OL should provide an editing interface to these links but directly edit in Wikidata via OAuth.

[LeadSongDog]

Second step could be to flesh out identifier lists and classification lists in the OL edition records using harvests from wikidata. This opens the door to finding other (non-IA) online-access copies.

[xayhewalo]

I think the steps are our good start but should be more granularly delineated. @hornc Your insight would be valuable in this thread.

[tfmorris]

@nichtich I'm surprised Open Library subject got approved as a Wikidata property. I recommend we discourage it's use since OL Subjects are a mess and going to change when we get around to either normalizing them or internationalizing them (or both). It has less than 600 uses now versus ~207,000 for the Open Library ID property.

@guyjeangilles I won't object if you want to break this into 5+ tickets, but that task could also be left until someone's ready to work on it in the spirit of Agile's just in time planning.

One thing I left off the original list was harvesting author birth & death dates, profession, AKAs, etc to help with disambiguation and photos for authors who don't have them.

[xayhewalo]

@tfmorris I'm not apposed to adopting Agile planning in the future, but considering our habit of leaving issues unattended for years, I'm assigning @hornc for the time being per slack discussion.

[RayBB]

Adding wikidata ids to works is blocked by #1797

[RayBB]

The first steps are awaiting review in #8236
I also have this handy wiki page with some idea and I've added yours there.

[tfmorris]

The first steps are awaiting review in #8236

So apparently that got closed and replaced by #9130 without linking it to this issue so that people could comment.

I also have this handy wiki page with some idea and I've added yours there.

Why use a wiki page instead of a series of sub-issues linked to this master issue (ie epic) so that they can be commented on? There's also apparently another secret version hiding here - https://docs.google.com/document/d/1-xAija9Pfhtwc-wCAgERHBx6GvFv40Hb-nfHd9f6SPc/edit

[mekarpeles]

@tfmorris just wanted to comment in that I don't think anyone is trying to make things secret or hidden and I don't imagine the phrasing makes folks feel particularly good / valued -- I observe people being generous with their time and working hard to be on the same team, contribute meaningfully to the project, and move things forward. We're a small community, there's a lot to do, and it can be hard to do everything to everyone's satisfaction. As a result, I think constructive criticism is pretty essential.

I value your passion and commitment to the community and appreciate that you invest the time and energy to voice your opinion. And at the same time, as per our code of conduct, I humbly and kindly request we try to make an effort to keep our critiques constructive and our language welcoming. It's more important that we feel comfortable and good about contributing as a team than getting everything right and I think we're more likely to respond well to criticism when it's delivered out of genuine care. Thanks for caring :)

[tfmorris]

@mekarpeles Just stumbled across this as I was revisiting needs to be done with Wikidata. Thank you for the reminder -- and thank you also for recognizing how generous we all are with our time.

I would humbly and constructively suggest that Github issues remain the focus, as they historically has been, rather than diverting to wikis, Google Docs, or other less accessible channels, and that new issues / PRs link to the relevant issues that they complement / address. You didn't comment on the main thrust of my suggestion in the previous reply, so I'm unsure as to whether this is something that you support or disagree with.

Thank you for recognizing that I continue to care despite having contributed for decades with zero thanks -- and Happy New Year!

[tfmorris]

Analytics review for Wikidata engagement in #10294

[mekarpeles]

@tfmorris, without sarcasm, I am grateful that you are so generous with your time. I'll reiterate that it's clear to me how much you care, that you have indeed invested a lot of time trying to help, and I take pride in both of these aspects of you. Furthermore, I feel bad if it hasn't felt like our thanks is reaching you. I want to at least dignify my own gratitude for you by pulling up a few cases where your efforts have been met with appreciation:

I agree with you, keeping github issues focused is also important to me and my goal.

There is no but.

Sometimes it takes time to figure out the right processes. We've tried wikis, Google Docs, or other channels help us brainstorm, plan, and have conversations in ways that ultimately we've hoped will help us keep github more focused. We try hard to make all of our documents public and link to them in issues when we use them. With 800 issues opened by a whole community of people, it can be difficult to both keep the quality of every issue high and also keep track of every effort so we've had to make some painful compromises and definitely made mistakes.

Sometimes good faith side-conversations are necessary to make sure everyone feels good about how things are going.

It appreciate it may feel like a distraction for us to have this type of conversation via a github issue, though I warmly extend that this type of good faith conversation we're having is also one I welcome on our Tuesday community calls, and that many contributors are having similar conversations on slack each week. I understand slack is a less accessible channel (yes, we could consider discord or matrix), yet these conversations often occur on slack for the same reason you're describing: so we ultimately can do more focused work on github.

I too would love for us to be able to use Github in a focused way. I more strongly feel that we have been given the trust of the open library community to stay true to the values of the code of conduct the community advocated for, and I intend to. I believe it's central to how we continue to be able to focus and work together effectively and civilly on Open Library.

Very respectfully, I am a human being that matters and I'd also love to not have to spend my time coopting github issues to address matters of code of conduct. It doesn't feel good to me when (a) I receive messages that other people may feel disrespected, (b) when I distract people from their work, and (c) that my own time needs to be spent this way.

Project management for Wikidata Integration

Regarding project management, my apologies re: #710 (comment) that #8236 was closed without linking to #9130. I own this failure for not doing a better job at ensuring the community has access to all the procedures staff typically tries to follow. @RayBB has similarly been working incredibly hard as a volunteer lead, I think he's been doing an exceptional job helping unblock the community with PR reviews, and I've found him to be very receptive to suggestions and feedback. Since ultimately PRs are triaged by me and staff, I will try to step up to make sure we do the right checks upfront to support both of you (@tfmorris and @RayBB) moving forward, so I thank you for bringing up these opportunities. Also, thank you @RayBB for trying to pull all our wikidata notes in one place. It seems like there may still be an opportunity to take our wiki + google docs + github issue notes and put them in one place... Github issues has the disadvantage of being sequential / comment based and makes it quite hard to collaboratively project roadmap.

I think really what may have been helpful is a call between stakeholders to converge on which specific Wikidata Integration efforts we want to try.

Back to the technical

• I agree with Integrate Wikidata  #710 (comment) that our subjects as is need a major rethink and aren't a great candidate for wikidata (in my humble opinion). Some subject perhaps could be defined based on Library of Congress classifications

• I'm supportive of us continuing to use author and work wikidata IDs where possible.

• I think IDs may make most sense as a starting point and dynamically pulling in data from Wikidata + caching them (and perhaps allowing librarians to choose from this data, as I believe @sbwhitt @davidscotson @cdrini and @RayBB have tried prototyping in the past).

• One of the difficult aspects of importing Wikidata keys and values into Open Library is figuring out the schema changes (e.g. series @cdrini) and also how to handle conflicting data (which system wins @hornc).

Thank you