Migrating from WikidataIntegrator#

WikibaseIntegrator started as a fork of WikidataIntegrator, whose last release is 0.9.30 (June 2023). The concepts are the same (entities, data types, references, qualifiers, the fast run mode), but the API was rewritten in v0.12: there is no compatibility layer, so a WikidataIntegrator script has to be adapted. This page maps the WikidataIntegrator 0.9.30 API to WikibaseIntegrator.

Installation#

python -m pip uninstall wikidataintegrator
python -m pip install wikibaseintegrator

WikibaseIntegrator requires Python 3.11 or newer.

A script, before and after#

A bot updating an existing item, with WikidataIntegrator:

from wikidataintegrator import wdi_core, wdi_login

login = wdi_login.WDLogin(user='MyUsername@MyBot', pwd='<bot password>')

references = [[
    wdi_core.WDItemID(value='Q20641742', prop_nr='P248', is_reference=True),
    wdi_core.WDTime('+2020-02-08T00:00:00Z', prop_nr='P813', is_reference=True),
]]

data = [
    wdi_core.WDExternalID(value='50943', prop_nr='P351', references=references),
    wdi_core.WDItemID(value='Q7187', prop_nr='P31', references=references),
]

# The values of P351 replace the existing ones, the P31 value is appended (append_value)
item = wdi_core.WDItemEngine(wd_item_id='Q14911732', data=data, append_value=['P31'])
item.set_label('FOXP3', lang='en')
qid = item.write(login, edit_summary='Update from NCBI Gene')

The same bot with WikibaseIntegrator:

from wikibaseintegrator import WikibaseIntegrator, wbi_login
from wikibaseintegrator.datatypes import ExternalID, Item, Time
from wikibaseintegrator.wbi_config import config
from wikibaseintegrator.wbi_enums import ActionIfExists

# Wikimedia asks every tool to identify itself: https://foundation.wikimedia.org/wiki/Policy:User-Agent_policy
config['USER_AGENT'] = 'MyBot/1.0 (https://www.wikidata.org/wiki/User:MyUsername)'

login = wbi_login.Login(user='MyUsername@MyBot', password='<bot password>')
wbi = WikibaseIntegrator(login=login)

# No is_reference: a reference is a list of claims, references a list of references
references = [[
    Item(value='Q20641742', prop_nr='P248'),
    Time(time='+2020-02-08T00:00:00Z', prop_nr='P813'),
]]

item = wbi.item.get('Q14911732')
item.claims.add(ExternalID(value='50943', prop_nr='P351', references=references), action_if_exists=ActionIfExists.REPLACE_ALL)
item.claims.add(Item(value='Q7187', prop_nr='P31', references=references))  # APPEND_OR_REPLACE by default
item.labels.set('en', 'FOXP3')
item = item.write(summary='Update from NCBI Gene')
qid = item.id

The main differences:

The entity is loaded first (wbi.item.get(), or wbi.item.new() for a new item), then modified, then written. Nothing is sent to the instance before write().
write() returns the written entity, not its ID: the ID is item.id.
The login is given once to WikibaseIntegrator(login=...) (or to each write(login=...)).
The arguments are named: labels.set(language, value) takes the language first, set_label(label, lang) took it last.

Entities#

WikidataIntegrator	WikibaseIntegrator
`WDItemEngine(wd_item_id='Q42')`	`wbi.item.get('Q42')`
`WDItemEngine(new_item=True)`	`wbi.item.new()`
`WDItemEngine(data=data)` without `wd_item_id`	No automatic search, see Finding an item from its data
`WDItemEngine(item_data=json)`	`wbi.item.new().from_json(json)`
`WDItemEngine(..., search_only=True)` then `update(data)`	`wbi.item.get()` then `item.claims.add(data)`: nothing is written before `write()`
`item.write(login, edit_summary='...', bot_account=True)`	`item.write(login=login, summary='...', is_bot=True)`
`write(..., entity_type='property', property_datatype='string')`	`wbi.property.new(datatype='string')` then `write()`
`item.get_wd_json_representation()`	`item.get_json()`
`item.lastrevid`, `item.wd_item_id`	`item.lastrevid`, `item.id`
`item.get_label('en')`, `set_label('Label', lang='en')`	`item.labels.get('en').value`, `item.labels.set('en', 'Label')`
`item.get_description('en')`, `set_description(...)`	`item.descriptions.get('en').value`, `item.descriptions.set('en', '...')`
`item.get_aliases('en')`	`[alias.value for alias in item.aliases.get('en')]`
`item.set_aliases(['A', 'B'], lang='en')` (append)	`item.aliases.add('en', ['A', 'B'])`
`item.set_aliases(['A', 'B'], lang='en', append=False)`	`item.aliases.set('en', ['A', 'B'], action_if_exists=ActionIfExists.REPLACE_ALL)`
`item.get_sitelink('enwiki')`, `set_sitelink(site, title, badges)`	`item.sitelinks.get('enwiki')`, `item.sitelinks.set(site, title, badges)`
`WDItemEngine.generate_item_instances(['Q1', 'Q2'])`	`wbi_helpers.generate_entity_instances(['Q1', 'Q2'])`
`sdc_core.WDItemEngine` (Commons)	`wbi.mediainfo.get('M75908279')`, see Structured Data on Commons

WikibaseIntegrator also reads and writes properties (wbi.property), lexemes with their forms and senses (wbi.lexeme) and MediaInfo entities (wbi.mediainfo). See Entities.

Data types#

The data types lose their WD prefix and are imported from wikibaseintegrator.datatypes. Their arguments are named:

WikidataIntegrator	WikibaseIntegrator
`WDString(value, prop_nr)`	`String(value=..., prop_nr=...)`
`WDItemID`, `WDProperty`, `WDLexeme`, `WDForm`, `WDSense`	`Item`, `Property`, `Lexeme`, `Form`, `Sense`
`WDExternalID`, `WDUrl`, `WDCommonsMedia`, `WDMath`	`ExternalID`, `URL`, `CommonsMedia`, `Math`
`WDGeoShape`, `WDTabularData`, `WDMusicalNotation`	`GeoShape`, `TabularData`, `MusicalNotation`
`WDMonolingualText(value, prop_nr, language='en')`	`MonolingualText(text=..., prop_nr=..., language='en')`
`WDQuantity(value, prop_nr, upper_bound, lower_bound, unit)`	`Quantity(amount=..., prop_nr=..., upper_bound=..., lower_bound=..., unit=...)`
`WDTime(time, prop_nr, precision=11, timezone, calendarmodel)`	`Time(time=..., prop_nr=..., precision=WikibaseTimePrecision.DAY, timezone=..., calendarmodel=...)`
`WDGlobeCoordinate(latitude, longitude, precision, prop_nr, globe)`	`GlobeCoordinate(latitude=..., longitude=..., precision=..., prop_nr=..., globe=...)`
`WDEDTF`, `WDLocalMedia`	`EDTF`, `LocalMedia`, from `wikibaseintegrator.datatypes.extra`
—	`EntitySchema`

The other arguments:

WikidataIntegrator	WikibaseIntegrator
`is_reference=True`, `is_qualifier=True`	Removed: the same classes are used for claims, qualifiers and references
`references=[[WDItemID(..., is_reference=True)]]`	`references=[[Item(...)]]`, or a `References` object
`qualifiers=[WDString(..., is_qualifier=True)]`	`qualifiers=Qualifiers()` with `qualifiers.add(String(...))`, or a list of claims
`rank='preferred'`	`rank=WikibaseRank.PREFERRED`
`snak_type='somevalue'`, `snak_type='novalue'`	`snaktype=WikibaseSnakType.UNKNOWN_VALUE`, `snaktype=WikibaseSnakType.NO_VALUE`
`check_qualifier_equality=False`	No equivalent: two claims are equal when they have the same property, value and qualifiers
`stat.get_value()`, `stat.get_prop_nr()`	`claim.mainsnak.datavalue['value']`, `claim.mainsnak.property_number`

The enumerations (WikibaseRank, WikibaseSnakType, WikibaseTimePrecision, ActionIfExists…) are in wikibaseintegrator.wbi_enums. See Data types.

Writing statements#

WikidataIntegrator took all the statements in the data list of the constructor and merged them with the item. WikibaseIntegrator adds them with item.claims.add(claims, action_if_exists=...), where action_if_exists says what to do with the existing statements of the same property:

WikidataIntegrator	WikibaseIntegrator
A property in `data` (the default): its other values are removed	`ActionIfExists.REPLACE_ALL`
A property in `append_value`: the value is added if missing	`ActionIfExists.APPEND_OR_REPLACE` (the default of `claims.add()`)
A data type with an empty value (`WDString('', 'P31')`): the property is removed	`item.claims.remove('P31')`
`WDFunctionsEngine.delete_statement(statement_id, revision, login)`	`item.claims.get('P31')[0].remove()` then `write()`, or `wbi_helpers.remove_claims(claim_id, login=login)`
`update(data, append_value=[...])`	More `item.claims.add()` calls

The reference handling is different. WikidataIntegrator merged the references of a statement already present following global_ref_mode (KEEP_GOOD by default), and kept the statements with “good” references (keep_good_ref_statements, good_refs). WikibaseIntegrator has no notion of good references, and its actions handle the references of a statement already present (same property, value and qualifiers) as follows:

WikidataIntegrator `global_ref_mode`	WikibaseIntegrator
`STRICT_OVERWRITE`	`APPEND_OR_REPLACE`: the references of the new claim, if it has some, replace the existing ones
`STRICT_KEEP_APPEND`	`MERGE_REFS_OR_APPEND`: the references missing from the existing statement are added
`STRICT_KEEP`	`REPLACE_ALL` or `KEEP` leave the existing statement as it is
`KEEP_GOOD`, `CUSTOM` (`ref_handler`)	No equivalent: get the existing statement with `item.claims.get()` and edit its `references`

REPLACE_ALL keeps the statements already present as they are, references included. To remove the other values of a property and add the new references to the statements already present, add the claims twice:

item.claims.add(data, action_if_exists=ActionIfExists.REPLACE_ALL)  # remove the other values of the properties
item.claims.add(data, action_if_exists=ActionIfExists.MERGE_REFS_OR_APPEND)  # add the new references

Finding an item from its data#

Without wd_item_id, WikidataIntegrator searched the item holding the values of data for the properties with a distinct-values constraint (core_props, core_prop_match_thresh), and created a new item if none was found. WikibaseIntegrator doesn’t search automatically: find the item yourself, for example with the fast run container, which resolves the item from a unique value with a SPARQL query:

from wikibaseintegrator import wbi_fastrun
from wikibaseintegrator.datatypes import ExternalID, Item

base_filter = [ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')]
frc = wbi_fastrun.get_fastrun_container(base_filter=base_filter)

qids = frc.get_entities(claims=[ExternalID(value='50943', prop_nr='P351')])
item = wbi.item.get(qids[0]) if qids else wbi.item.new()

or with the search of the MediaWiki API, which supports the haswbstatement keyword on Wikidata:

from wikibaseintegrator import wbi_helpers

results = wbi_helpers.fulltext_search('haswbstatement:P351=50943')

Fast run#

WikidataIntegrator	WikibaseIntegrator
`WDItemEngine(data=data, fast_run=True, fast_run_base_filter={'P351': '', 'P703': 'Q15978631'})`	`item.claims.add(data)` then `item.write_required(base_filter=[ExternalID(prop_nr='P351'), Item(prop_nr='P703', value='Q15978631')])`
`item.require_write`	The value returned by `item.write_required()`
`fast_run_use_refs=True`	`write_required(..., use_references=True)`
`fast_run_case_insensitive=True`	`write_required(..., case_insensitive=True)`

The base filter is a list of data type instances instead of a dictionary: a value is a required value, no value means any value. It also supports property paths. See Fast run.

Login#

WikidataIntegrator	WikibaseIntegrator
`WDLogin(user='User@Bot', pwd='<bot password>')`	`wbi_login.Login(user='User@Bot', password='<bot password>')`
`WDLogin(user, pwd, use_clientlogin=True)`	`wbi_login.Clientlogin(user=..., password=...)`
`WDLogin(consumer_key=..., consumer_secret=...)` then `continue_oauth()`	`wbi_login.OAuth1(consumer_token=..., consumer_secret=...)` then `continue_oauth()`
—	`wbi_login.OAuth2(consumer_token=..., consumer_secret=...)` for an owner-only consumer, the recommended way for a bot
—	`wbi_login.OAuth1(consumer_token=..., consumer_secret=..., access_token=..., access_secret=...)` for an owner-only OAuth 1.0a consumer

See Login.

Helpers#

WDFunctionsEngine and the static methods of WDItemEngine become functions of wikibaseintegrator.wbi_helpers:

WikidataIntegrator	WikibaseIntegrator
`execute_sparql_query(query, prefix, endpoint, user_agent)`	`wbi_helpers.execute_sparql_query(query, prefix, endpoint, user_agent)`
`execute_sparql_query(..., as_dataframe=True)`	No equivalent: build the dataframe from `results['results']['bindings']`
`mediawiki_api_call_helper(data, login, ...)`	`wbi_helpers.mediawiki_api_call_helper(data, login=login, ...)`
`get_wd_search_results('Douglas Adams', language='en')`	`wbi_helpers.search_entities('Douglas Adams', language='en')`
`merge_items(from_id, to_id, login_obj, ignore_conflicts='description')`	`wbi_helpers.merge_items(from_id, to_id, login=login, ignore_conflicts=['description'])`
`get_rdf(qid, format='turtle')`	`wbi_helpers.download_entity_ttl(qid)`, or `entity.download_entity_ttl()`
`delete_item(item, reason, login)`: a request on Wikidata:Requests for deletions	No equivalent. `wbi_helpers.delete_page()` deletes a page directly, with the delete right
`WDItemEngine.wikibase_item_engine_factory(mediawiki_api_url, sparql_endpoint_url)`	`wbi_config['MEDIAWIKI_API_URL']`, `wbi_config['SPARQL_ENDPOINT_URL']`, see Installation and configuration
`WDItemEngine.setup_logging()`, `WDItemEngine.log()`	The standard `logging` module, see Installation and configuration

WikibaseIntegrator has more helpers (redirects, lexeme forms and senses, constraint checks, value parsing, EditGroups…): see Helpers.

Configuration#

wdi_config.config becomes wikibaseintegrator.wbi_config.config, with the same keys, except:

USER_AGENT_DEFAULT becomes USER_AGENT. Wikimedia sites require a User-Agent identifying your tool.
CONCEPT_BASE_URI is gone: the concept URIs are built from WIKIBASE_URL.
MAXLAG is gone: maxlag is an argument of the API calls (mediawiki_api_call_helper(..., maxlag=5), 5 by default).
The retries: write() and the API calls retry 20 times by default (max_retries), instead of 1000.

Without an equivalent#

These parts of WikidataIntegrator have no equivalent in WikibaseIntegrator:

The search of an item from its data (core_props), see Finding an item from its data.
The good references (good_refs, keep_good_ref_statements, KEEP_GOOD) and the custom reference handlers (ref_handler, the ref_handlers module).
rollback(), get_linked_by() (use a SPARQL query), count_references(), get_reference_properties(), get_qualifier_properties().
The ShEx checks (check_entity_schema(), check_shex_conformance(), extract_shex()).
wdi_helpers (PubmedItem, PublicationHelper, Release, try_write, prop2qid, id_mapper…), wdi_rdf and MappingRelationHelper.