The relational database model is well grounded in theory and well supported by the software industry. Relational database management systems (RDBMS) replaced legacy hierarchical and network database products in nearly every enterprise and became the mainstream. Object-oriented DBMS have not taken the leading role from relational systems, even after object-oriented programming became the norm, and in spite of the so-called \u201cimpedance mismatch\u201d perceived when one needs to store nested objects into a collection of flat tables.<\/p>\n
After resisting the object-oriented wave, relational databases became so dominant that they seemed like the only game in town, or at least the only game in most shops. But in the last five years, Google published a paper on BigTable (Chang, 2006<\/a>), the proprietary, distributed non-relational database behind many of its online properties. Then Amazon.com revealed Dynamo, the non-relational database that powers some of their AWS cloud-based services (Decandia, 2007<\/a>). Facebook open-sourced Cassandra, the distributed non-relational database created to enable its crucial in-box search feature. Cassandra is now also used by Twitter, Digg, Rackspace and Cisco, among many others, and has such a vibrant community that it became a top-level project of the Apache Foundation in 2010, less than three years after its initial public release (Apache, 2011a<\/a>). These developments are part of a trend that became known, in 2009, as the \u201cNoSQL” movement. NoSQL stands for \u201cNo SQL\u201d or \u201cNot Only SQL\u201d, depending on who you ask; but everyone agrees it’s not really about the SQL language, but about seeking alternative answers to problems that are not amenable to relational solutions.<\/p>\n Meanwhile, thousands of small to medium libraries in developing countries have been oblivious to all this and continue using their ISIS databases for daily operations. ISIS is a family of non-relational database systems with a history that goes back to the 1970’s. It was developed by UNESCO \u2013 the United Nations Educational, Scientific and Cultural Organization \u2013 specifically for bibliographic data, and it is still officially distributed and widely used (Hopkinson, 2005<\/a>).<\/p>\n ISIS had a very positive impact, allowing libraries with limited resources to computerize their catalogs on simple, stand-alone PCs. Some ISIS products became Web-enabled, but after that, progress has been very slow in the last 10 years. Meanwhile, a couple of new Open Source NoSQL databases present a possible migration path for ISIS users, a path that will not require wholesale normalization of decades of bibliographic data. This paper describes how BIREME\/PAHO\/WHO, a digital library that is part of the Pan American Health Organization, and a large-scale user of ISIS databases, has studied the viability of migrating one of its main databases to Apache CouchDB, a modern document database.<\/p>\n The rest of the paper is structured as follows. Section 2<\/a> discusses data models for bibliographic records, explains why it is tempting for librarians to look beyond the relational model and compares the structure of MARC records to the semistructured data model formalized in the 1990’s. Section 3<\/a> briefly presents ISIS, then compares two recent products that support semistructured records, CouchDB and MongoDB, and shows our criteria for choosing among them. Section 4<\/a> discusses different formats for representing ISIS records in CouchDB, then describes the tool we developed for converting records and the process used to load them into CouchDB. Section 5<\/a> talks about how indexing works in ISIS and CouchDB, and exemplifies queries on the latter. Finally, Section 6<\/a> summarizes what we have done, our conclusions and ongoing work.<\/p>\n One of the defining characteristics of the relational data model, the First Normal Form (1NF), dictates that attributes must be atomic. In other words, in the industry-standard flat relational model1<\/a><\/sup>, fields cannot be structured into subfields nor contain multiple values.<\/p>\n On the other hand, multivalued fields are useful and common for representing bibliographic data. Books often have more than one author and cover multiple subjects. A publisher, while logically a single attribute of a book record, has attributes of its own, like a name and a place. It is therefore useful to split a publisher field into parts. That is why MARC supports subfields and multivalued fields through repeating tags (similar to XML). The MARC structure, or \u201cempty container\u201d2<\/a><\/sup> carried over to the ISO-2709 standard, which describes a generic data interchange format allowing applications to attach any meaning to the tags and subfield markers. Here we will refer to the \u201cISO-2709 data model\u201d as a generalization of the data model implied by the MARC record structure.<\/p>\n Of course, library data is often stored in relational databases, but normalization means that the description of a single item is spread over multiple records in several tables. That may work well within the context of one library, but when exchanging data, it is more practical to have just one record per item. That is one reason why MARC is still with us, and why XML became so important in a world dominated by relational databases.<\/p>\n If the ISO-2709 data model does not adhere to the flat relational model, then what is the theory behind it? At first there was none, but in the mid-1990s, while XML was still under development, researchers from UPenn, Stanford, AT&T Labs and the INRIA French R&D agency conceived the semistructured3<\/a><\/sup> data model (Abiteboul, 1999<\/a>). Their work supports reasoning about MARC and ISO-2709 records but also about the richer data models of XML and JSON4<\/a><\/sup> documents.<\/p>\n A useful definition by Dan Suciu, a pioneer of semistructured data research, appears in the \u201cEncyclopedia of Database Systems\u201d (Suciu, 2009<\/a>):<\/p>\n The semi-structured data model is designed as an evolution of the relational data model that allows the representation of data with a flexible structure. Some items may have missing attributes, others may have extra attributes, some items may have two or more occurrences of the same attribute. The type of an attribute is also flexible: it may be an atomic value or it may be another record or collection. Moreover, collections may be heterogeneous, i.e., they may contain items with different structures. The semi-structured data model is a self-describing data model, in which the data values and the schema components co-exist. <\/p><\/blockquote>\n Thanks to the semistructured data model, we don’t need to feel ashamed of our denormalized bibliographic records any more. Even better, we have a framework to evaluate databases of the current NoSQL crop with regard to our needs as caretakers of denormalized datasets.<\/p>\n The book \u201cData on the Web: From Relations to Semistructured Data and XML\u201d (Abiteboul, 1999<\/a>), introduces a notation for representing semistructured data. It is called \u201cssd-expression\u201d and it looks like this:<\/p>\n Except for the repetition of the The ISO-2709 record format influenced the record structure of the ISIS family of databases, introduced by UNESCO in the 1970s. ISIS records follow the structure of the ISO-2709 standard, and most ISIS systems import and export to that format. Here is the same record of the examples above, as seen in a common ISIS display format:<\/p>\n In an ISIS record the field names are replaced by numeric tags. The \u00ab\u00bb delimiters are not part of the content, but a feature of the display format. Here there are two occurrences of tag #30, the e-mail field. Field #10 is split into two subfields (“f” and “l”) by special markers. Instead of the subfield delimiter control character used in MARC systems, ISIS uses the circumflex accent: ^ (ASCII 94). A subfield identifier is case-insensitive and consists of just one ASCII letter or digit. The ISIS data model is simpler than that of ISO-2709 because it does not have field indicators5<\/a><\/sup> and subfields are not repeatable6<\/a><\/sup>.<\/p>\n Clearly the ISIS data model is not as flexible as the general semistructured data model. Given the syntax of subfield markers, ISIS fields are limited to one level of nesting only. In other words, borrowing from the XML jargon: subfields cannot have child elements, just character data. Also, an ISIS field may have mixed content7<\/a><\/sup>: some unmarked text may appear before the first subfield marker. For example, in this field:<\/p>\n The main title, “Lords of Finance” is not preceded by a subfield marker. The semistructured data model, as described by Abiteboul et. al., has no concept of mixed content like XML has. So we will call that part \u201cthe main subfield\u201d and pretend it is preceded by an implied “^_” (underscore) subfield marker whenever we need to refer to it in code. With this simple arrangement, any ISIS record can be represented as an ssd-expression or a JSON record.<\/p>\n In 1985, Giampaolo Del Bigio ported UNESCO’s CDS\/ISIS mainframe database system to PC\/DOS (Lopes, 2010<\/a>). The result, called MicroISIS, was distributed free of charge and became the de facto<\/i> standard for computerized library catalogs in developing countries. Its Windows version, WinISIS is still distributed by UNESCO and remains widely used in small to medium libraries (Hopkinson, 2005<\/a>). Beyond local OPACs, ISIS is also used by two large regional bibliographic databases in Latin America and the Caribbean: SciELO (Scientific Electronic Library Online) and LILACS (Latin American and Caribbean Health Sciences index), both built by BIREME\/PAHO\/WHO, which is a specialized center of the Pan American Health Organization located in S\u00e3o Paulo, Brazil. Together, SciELO and LILACS routinely handle millions of bibliographic records using CISIS, a custom version of ISIS developed by BIREME\/PAHO\/WHO. MicroISIS, WinISIS, CISIS and the more recent J-ISIS from UNESCO are all interoperable and form the ISIS family of database systems.<\/p>\n The legacy ISIS codebases are showing their age, and after years of closed-source development, none of the derivations have managed to become successful Open Source projects. Meanwhile, the NoSQL movement has highlighted some Open Source non-relational databases implementing variations of the semistructured data model. This has motivated BIREME\/PAHO\/WHO to look for alternatives.<\/p>\n Among the recent crop of NoSQL systems, two products stand out for implementing the semistructured data model in a way that matches the operational needs of BIREME\/PAHO\/WHO: Apache CouchDB and MongoDB (Apache, 2011b<\/a>, MongoDB.org, 2011<\/a>). Their data model is not as flat as the key-value stores (optimized for retrieving blobs given a primary key), nor as deep as graph databases (designed to allow general queries over paths of nested objects). What they offer is somewhere in between: JSON-like records allowing nested structures, and expressive query languages to index and retrieve those records. Those rich records are called \u201cdocuments\u201d. CouchDB and MongoDB call themselves \u201cdocument databases\u201d.<\/p>\n In the case of CouchDB, the document format is JSON. MongoDB uses BSON, a binary format inspired by JSON but offering more data types, such as int32, datetime, and even a type called \u201cJavaScript code w\/ scope\u201d (Dirolf, 2010<\/a>). JSON can be trivially converted to BSON, but the reverse may not be so easy, depending on the data. While the difference between those formats can be considered an implementation detail, they reflect different priorities.<\/p>\n MongoDB is optimized for fast updating, by aggressively caching writes and by overwriting updated records in place whenever possible: the BSON structure is designed to allow updating specific fields of an existing document. The flip-side of this optimization is that a software crash can cause data loss, and a minimum deployment of two servers in master-slave configuration is recommended if durability is required8<\/a><\/sup>. In contrast, CouchDB is fault-tolerant: it only appends to the database file on disk, and that is an atomic operation in modern operating systems. As a result, the database file is always consistent in the event of a software crash, and backups can be made while the system is running. The drawbacks of the append-only design are the need for periodic database compaction \u2013 a time-consuming batch operation \u2013 and slower updates.<\/p>\n CouchDB was influenced by Lotus Notes, a networked, collaborative application suite designed to support users who are often off-line. So, CouchDB allows master-master replication, that is, synchronization between peer nodes which have received inserts and updates independently. MongoDB supports only master-slave replication: only one node receives updates and inserts, and replicates to the slaves.<\/p>\n While CouchDB can be used in large clusters, it is also well-suited as a database for small desktop apps. For that reason it comes pre-installed on Ubuntu Linux since 2009. It even runs on Android mobile devices (a free, one-click installer can be found in the Android Marketplace). Because CouchDB supports HTTP and JSON natively, and can run JavaScript procedures on the server, applications can be developed without any middleware: browsers communicate directly with the database, and all the logic is written in JavaScript.<\/p>\n The informative post \u201cComparing Mongo DB and Couch DB\u201d was written by Dwight Merriman, CEO of 10gen, the company behind MongoDB (Merriman, 2010<\/a>). Given the characteristics of these systems, we decided to use CouchDB for our LILACS experiments for these reasons:<\/p>\n However, we do envision scenarios in which we would like to use both MongoDB and CouchDB. For example, the canonical store of bibliographic data could be CouchDB, while a cluster of MongoDB instances could be used for user-provided content, tracking, recommendations and other features demanding faster database writes.<\/p>\n The ISIS data model, a subset of the ISO-2709 model, is not as expressive as the JSON data model. There are several ways to represent the same ISIS record in JSON. Consider this abridged example of a LILACS record:<\/p>\n Tag #2 is the LILACS identifier, a non-repeating, numeric field, and tag #10 is an author field in an analytic (article-level), record. The LILACS data dictionary (BIREME, 2008<\/a>) defines #10 as a repeating field. In addition, it is composed of the following subfields:<\/p>\n<\/a>2. Data models for bibliographic records<\/h2>\n
\r\n {name: {first: \"Lewis\", last: \"Carroll\"},\r\n tel: 5553457,\r\n email: \"cld@ox.ac.uk\",\r\n email: \"lcarroll@pobox.com\"\r\n }\r\n<\/pre>\nemail<\/code> key, it’s very similar to JSON (Crockford 2006a<\/a>). The presence of keys describing fields like name<\/code>, first<\/code> etc., and the storage of those keys alongside the data, is what the encyclopedia entry means by coexisting data values and schema components (Suciu, 2009<\/a>). It is a characteristic shared by ISO-2709, XML, and JSON (with the limitation that ISO-2709 fields are identified by tags composed of three alphanumeric ASCII characters, though most applications only use numeric tags). The repeated email<\/code> tag could be represented similarly in XML and in ISO-2709, but in JSON the keys must be unique within an object structure, so multivalued fields are often represented as lists of values:<\/p>\n\r\n {"name": {"first": "Lewis", "last": "Carroll"},\r\n "tel": 5553457,\r\n "email": ["cld@ox.ac.uk", "lcarroll@pobox.com"]\r\n }\r\n<\/pre>\n\r\n 10 \u00ab^fLewis ^lCarroll\u00bb\r\n 20 \u00ab5553457\u00bb\r\n 30 \u00abcld@ox.ac.uk\u00bb\r\n 30 \u00ablcarroll@pobox.com\u00bb\r\n<\/pre>\n
\r\n 18 \u00abLords of Finance^bThe Bankers Who Broke the World\u00bb\r\n<\/pre>\n
<\/a>3. Semistructured database systems<\/h2>\n
3.1. The ISIS family<\/h3>\n
3.2 CouchDB and MongoDB<\/h3>\n
\n
<\/a>4. Loading ISIS records into CouchDB<\/h2>\n
4.1. Alternative representations of ISIS records as JSON documents<\/h3>\n
\r\n 1 \u00abBR1.1\u00bb\r\n 2 \u00ab538886\u00bb\r\n 4 \u00abLILACS\u00bb\r\n 4 \u00abLLXPEDT\u00bb\r\n 5 \u00abS\u00bb\r\n 6 \u00abas\u00bb\r\n 8 \u00abInternet^ihttp:\/\/\u2026\/imageBank\/PDF\/v3n3a04.pdf?aid2=168&\u2026\u00bb\r\n 10 \u00abKanda, Paulo Afonso de Medeiros^1University of S\u00e3o Paulo\r\n ^2School of Medicine^3Cognitive Disorders of Clinicas\r\n Hospital Reference Center^pBrasil ^cS\u00e3o Paulo^rorg\u00bb\r\n 10 \u00abAnghinah, Renato^1University of S\u00e3o Paulo^2School of\r\n Medicine ^3Cognitive Disorders of Clinicas Hospital\r\n Reference Center ^pBrasil^cS\u00e3o Paulo^rorg\u00bb\r\n 12 \u00abThe Clinical use of quantitative EEG in cognitive disorders\r\n ^ien\u00bb\r\n 12 \u00abA utiliza\u00e7\u00e3o cl\u00ednica do EEG quantitativo nos transtornos\r\n cognitivos^ipt\u00bb\r\n 30 \u00abDement. neuropsychol\u00bb\r\n 31 \u00ab3\u00bb\r\n 32 \u00ab3\u00bb\r\n 35 \u00ab1980-5764\u00bb\r\n<\/pre>\n
![\"Futon](\"\/media\/issue13\/ramalho\/fig1-futon-500x422.png\")
<\/a><\/div>\n![\"Futon](\"\/media\/issue13\/ramalho\/fig2-tempview-500x399.png\")
<\/a><\/div>\n