Introducing

BIB MAGIC IMPORTER

Bib Magic Importer is:

• Amazing
• Incredible
• Phenomenal
• Magnificent
• Spectacular

No actual Google synonyms were harmed during the making of this slide

Allow me to show you why

Ongoing imports

Auto Electronic Scoping

1. Configure a list of participating libraries
2. Auto merge subfield 9's
3. Duplicate bibs from multiple vendors

Authority Records

SILOS

Designed for compartmentalized MARC records per import project(s)

Remote record retrieval

FTP / Cloud Library / MARCIVE HTTPS

Match existing records

1. Incoming MARC can scope-match to it's own set
2. But can be expanded to match any Evergreen bib
3. Duplicate bibs are deduped and 856's are merged

Auto MARC Manipulation

Configure to change incoming MARC automatically

Add fields

Remove fields/subfields

Replace fields/subfields

Also works with control fields

Sync $9's

For electronic imports: newly added libraries can be retro-actively added to existing 856's

Results reporting

1. Import jobs will email a start/finish report
2. Use Evergreen reporter to get more details

Checklist

You'll need access to the server

It's one Perl program

Checklist

Standard command

./bib_magic_importer.pl --config overdrive.conf

Sync $9's

./bib_magic_importer.pl --config overdrive.conf --syncnines

Force match on 901c only

./bib_magic_importer.pl --config overdrive.conf --match_901c

Config

The easy stuff

Config

recordsource

##########################################################
# Options are: FTP or folder or cloudlibrary or marcivehttps
# Note: cloudlibrary option enables the API connection to
# the remote server.
##########################################################
Examples:
recordsource = folder
recordsource = cloudlibrary
recordsource = FTP
recordsource = marcivehttps
                        

Config

Remote server stuff

Config

sourcename

###################################################################
# This is a way to keep track of the set of records in the
# database. This string value will be entered into
# config.bib_source if it doesn't exist. All imported
# records will be tagged with this string value as the source.
# NOTE: Auto record deduplication does not cross sources. Therefore,
# if you have multiple electronic MARC record imports, it could be
# a good idea to use the same source value for each such that the
# records can merge onto eachother. 856's are melted together each
# with the appropriate participant $9's.
##################################################################
Examples:
sourcename = overdrive
sourcename = electronic
                        

Config

sourcename

Config

bibtag

Think of it as a record sub-category

############################################################
# This value is appended to the tcn_source when bibs
# are imported. Later, this is referenced during removals.
# Only bibs with this same tag will be scoped when searching
# for removals. import_as_is will cause this setting to be ignored
############################################################
Examples:
bibtag = overdrive_advantage
bibtag = gov_docs
                        

Config

tcn_source_authority

This needs to (almost) always be "yes"

You might want to disable this if you are importing records that you exported from Evergreen

Config

participants

############################################################
# Comma separated list of shortnames for each of the
# branches/systems that will be included in the 856$9 values
############################################################
Examples:
participants = SYS1,SYS2
participants = BR1,BR2
participants = SYS1,BR2
                        

Config

merge_9s

#######################################################
# If there is a matching record in the database, and
# that matching record has an identical 856 with a matching
# $u, then we will merge all of the $9's together or not.
#######################################################
Examples:
# merge_9s = yes
merge_9s = yes
                        

Config

import_as_is

#######################################################
# Short circuit any code that would change the MARC
#
# You can optionally cause the software to do nothing
# to the marc and import as-is This is usually in
# conjuction with passing --match_901c to the script
# 
#######################################################
Examples:
# import_as_is = yes
import_as_is = yes
                        

Config

Filename matching

These options help the script understand which file(s) you would like it to process. And if a file should be treated as a "removal" instead of an "add".

All options are space delimited

Config

authority_link_script_cmd

#######################################################
# When importing authority records, we will automatically
# run the linking script but we need to know where
# it is and the cmd switches
#######################################################
Example (line breaks for presentation):

authority_link_script_cmd =
/openils/bin/authority_control_fields.pl
        --config /openils/conf/opensrf_core.xml
        --record
                        

Config

eg_staged_bib_overlay_dir

#######################################################
# If you use this script to import authority files,
# you will also need to have the "eg_staged_bib_overlay"
# script available. This is where you define the path
# to the root folder.
# git repo
# https://github.com/EquinoxOpenLibraryInitiative/migration-tools
#######################################################
Example:
eg_staged_bib_overlay_dir = /path/to/migration-tools
                                                    

Config

Email notifications

All options are comma delimited

erroremaillist = [email protected]
successemaillist = [email protected], [email protected]
alwaysemail = [email protected]
                        

MARC MANIPULATIONS

MARC MANIPULATIONS

Adding Fields

marc_edit_standard_1 = 'type' => 'add',
'def' => ['650','0','0','a','Whatever you want to add']
                

MARC MANIPULATIONS

Removing Whole Fields

marc_edit_standard_2 = 'type' => 'remove',
'def' => ['852','853','995']
                

MARC MANIPULATIONS

Conditionally remove fields

marc_edit_standard_3 = 'type' => 'remove',
'howmany' => 'all',
'def' => ['856_4_none_3', '856_1_1_a', '651_all_all_b']
                

MARC MANIPULATIONS

Removing subfields instead of whole fields

marc_edit_standard_4 = 'type' => 'removesubfield',
'howmany' => 'all',
'def' => ['756', 'q']
                

MARC MANIPULATIONS

Replace

marc_edit_standard_5 = 'type' => 'replace',
'howmany' => 'all',
'def' => ['856','same','same','y','Click for online content.']
                

MARC MANIPULATIONS

Control fields

marc_edit_control_1 = 'type' => 'replace', 'def' => ['008','24','o']
marc_edit_control_2 =
        'type' => 'remove',
        'howmany' => '1',
        'def' => ['007']
marc_edit_control_3 =
        'type' => 'replace',
        'def' => ['007','1','vd cvaizq']
                

Reporting

BIB Magic has it's own database schema: bib_magic

Reporting

Bib Magic Job

Each import creates a "job" which is assigned a sequential ID number

Reporting

Bib Magic Import Status

Each job contains bib with details in this table

Reporting

Bib Magic MARC Update

When a bib is overlayed, it's recorded here

Makes the bib update reversable

Reporting

Bib Magic Item Reassignement

If items are moved to another bib, it's recorded here

Makes the bib merge reversable

Reporting

Bib Magic BIB Merge

When two bibs are merged, it's recorded here

Makes the merge reversable

Reporting

Bib Magic Nine Sync