How Do We Do This?

OK, historian and digerati friends--how should I proceed with this project? We have an Excel spreadsheet inventory of 7000 items from a defunct museum, along with 7000 individual JPEG pictures of those items. How do we put these together, attaching an image to each spreadsheet item, in a useable format?

 Some background. From 1980 until some time in the late 1990s, there was a museum on the Fairchild Air Force base in eastern Washington. The museum was one of those labors of love, mostly powered by volunteers from the local military community. The focus was not only the base, but the military history of the Inland Northwest--so the collection  ranges from minnie balls picked up from 19th-century battlefields to a B-52 flight simulator. In the late 1990s the Air Force closed most base museums, including Fairchild. As the  artifacts were boxed  for storage, a quick inventory was created for the roughly 7000 items and a digital photograph was taken of each. The boxes went into storage at the Northwest Museum of Arts & Culture. Where they remain to this day:

MAC Director Forrest Rogers unveils some
of the ~300 boxes of the collection.

 My public history class has been tasked with surveying the collection and deciding what are some of the particularly interesting and noteworthy items. Obviously whatever we do will be only a start, and a professional conservator will have to come in at some point, but whatever we can do will help the owners of the collection as they work towards their plan of establishing a new aerospace museum in Spokane. After a conversation with board president Tobby Hatley, we decided creating a database of the spreadsheet and images would be the first order of business for my students and I.

My initial thought was to simply upload the spreadsheet to Google Docs, share with the class, add a column titled "images" and start dragging and dropping. I strongly suspect, however, that this is too large a data set to work with that way. So what do we do? We are experienced with Omeka from our work with Spokane Historical--can we simply convert a spreadsheet to an Omeka database in some fashion, or are are hand-creating each item? Is there another solution we should be examining--Drupal? Wordpress?

We are looking to create a working database and not (at this time) a digital project. We want to keep it quick and reasonably simple, so my students can move to the next phase--opening some of those boxes and locating the treasures of this collection.

How do we do this?

Mapping Spokane's Dead: A Pedagogical Experiment in Flash-mob Data Visualization

I taught Digital History last quarter. The course was a lot of fun as a dozen traditionally-trained MA students and I explored some of the new digital historical landscapes. The class is divided between readings discussions and a weekly "make" where we bust open some new digital tool and see what we can do with it. For our weekly make a few months ago we created, populated, and visualized a historical database in about an hour. This post is about what we did and how we did it.

I had a couple of different pedagogical goals.First, I wanted my students to understand the importance and power of constructing a database, rather than merely building a website. Second, I wanted them to explore one tool for doing so, Google Fusion Tables. Third,.I wanted to use some of the rich historical resources of my employer, the Washington State Archives, Digital Archives. Finally I wanted them to experience the difficulty, decisions and compromises of building a database and extracting metadata from handwritten historical documents.

Working with my grad student, the excellent Lee Nilsson, we chose the Spokane County Death Returns.1888-1907 as our data set. These records were interesting, the images of the death certificates and some metadata were already online, they represented a broad cross-section of the population of early Spokane, and they presented certain complications as well, in terms of handwriting and uneven data. Here is a sample death return from the collection, that of the unfortunate Owen J. Jones:

When the State Archives originally scanned and indexed these records, they chose to record as the metadata the first and last names, age at death, and place of death. This was a good start, but missed some data that was recorded on the death certificates and that historians would find important. So we added race, occupation, place of birth and cause of death to the metadata fields that we wanted to capture.

Then Nilsson created a Google Fusion table with our metadata fields and entered the information from a few death returns. Right away he realized that one problem the students would run into was transcribing the causes of death--things like phthisis (tuberculosis of the lungs) and morasumus (malnutrition) written in sometimes terrible 19th-century handwriting. A quick Google found us some lists of 19th century causes of death. Nilsson added about 150 names from the 1880 death index to the Google Fusion table and used color highlighting to organize the list in groups of ten names. I took the email addresses of the students in the class and gave them permission to edit the table.

The actual lesson took about an hour. We gave each student ten names and asked them to read the death certificates and to add the metadata to the table. Nilsson and I circulated in the classroom to help people out.   The names and dates went pretty smoothly. The students stumbled with causes of death at first, but the guides to 19th century causes of death cleared up most questions. A bigger problem was missing data. 1880 was the first year of death records in Spokane and the record keeping was erratic. Places of birth, causes of death, and other items were not always filled out.

Then came the experiment part--visualizing the data. My original inspiration for the project was the idea that we would produce a map of where in Spokane people had died. This did not work so well:



In retrospect the reasons are clear. A few of the death certificates gave street addresses, and Google was able to map these accurately. In other cases no place was given, or given only as Spokane. In some cases there was a location but the student transcribing was unable to read it or just did not bother. With better instruction and monitoring this map might have come out better.

Much more satisfying was this map of where the people were born. It nicely illustrates patterns of migration into 19th century Spokane. Don't miss the guy from China!

Google Fusion tables allow for other types of visualization as well. Here is a pie graph of causes of death:



Those are some mighty thin slices of pie! The chart does not tell us very much, except that typhoid and pneumonia were common. We would have been better off creating categories--contagious disease, accidents, infant death, etc.

You can also make bar graphs--here is one for the occupations of the deceased. The laboring classes seem to have had it bad in early Spokane, though you would need some demographic analysis to make any conclusions here:


Finally, here is the a frequency showing the distribution of deaths throughout the year. The table is interactive, you can click and drag to explore it. Notice anything odd? The Grim Reaper seems to have forgotten Spokane entirely some months:



The students were quite puzzled by this and came up with all sorts of reasons that several months could have passed without a death. Of course the most likely explanation is simply that the records for those months have gotten lost in the century since they were initially recorded.

All in all this pedagogical experiment was a great success. My students learned how digital sausage is made--the decisions that go into choosing what metadata to record and visualize, the challenges of working with hand-written 19th-century documents, the amount of pain-staking work that went into a data visualization.

How many photos have ever been taken?

How many photos have ever been taken? About 3.5 trillion according to Jonathan Good, in a fascinating piece. I particularly liked this graphic of the world's largest photo libraries:

Historic Photographs from the Spokane Public Library

Here is a great new resource for local history: Spokane Public Library - Northwest Room Digital Collections. "The Northwest Room of Spokane Public Library is pleased to introduce a digital collections page of photographs selected from collections in the Northwest Room. These collections emphasize the most frequently requested subjects in the Northwest Room – the homes, buildings, streets and activities in and around Spokane."

Remains of the Glover/Pioneer Block after the fire of 1889.

So far the collection contains about 350 items, and Northwest Room librarian Riva Dean says that the library plans to add "plenty more." The initial collection includes historic Spokane photographs in six categories: the Spokane Fire (1889), Spokane River, Spokane Views, Spokane Homes, Spokane Parks, Spokane Bridges, and Spokane Streets. Unfortunately the images are provided using ContentDM, which is the industry standard despite being a fairly stodgy piece of commercial software. It presents information adequately but does not allow users to interact with item to add data or offer corrections. (It would be nice, for example, to flag this item, which contains a picture of one Spokane park with a description of a different park, without having to go back to the library webpage and figure out who to email the correction.) It also does not allow users to easily save or export the images or to share them with friends via Facebook or other social media. (I extracted the image on this page using the Picnik extension on the Chrome browser.)

The first set of pictures are delightful, from scenes of the devastating 1889 fire to early street images to intriguing homes (anyone know where this one is?).

It is a shame the site is not more interactive because these are wonderful photographs and could easily become nodes of online discussion about Spokane history. Many have only partial information with them-the dates are unknown, the locations have been forgotten, etc. If there were a discussion area with each photo, patrons could probably help fill in a lot of the missing information.

The Spokane Public Library is doing a real service to its public by making these photographs so easily accessible. I look forward to watching them add to the collection. Of course if you are the impatient type you can just go to the Northwest Room of the Spokane Public Library, where they have a vast collection of primary resources for Spokane History.

A Post for Jerry Handfield

Jerry Handfield, the Washington State Archivist and one of my bosses, loves these Team Digital Preservation videos. DigitalPreservationEurope has created a series of these cartoons to help explain basic concepts of digital preservation. There are a half-dozen videos in the series so far, you can see the rest here.

Online Resources at the Washington State Historical Society

I began working on a post about the new Columbia magazine website, but along the way found a treasure trove of other history resources at the Washington State Historical Society. So I will begin with the magazine and branch out from there.

I see that Columbia: The Magazine of Northwest History, the journal of the Washington State Historical Society, has a new and improved website. Those of you who are familiar with Columbia know that it is a superb journal of popular history, with articles and book reviews that present recent scholarship to a popular audience. The website does not contain everything that has appeared in the journal but many of the articles are available in full text. (This article is excellent.)

There are many other online resources available from the WSHS, including lesson plans, featured collections, women's history resources, and information about visiting the WSHS locations in Tacoma and Olympia. The featured collections is an especially rich resource, and includes digital collections of maps, American Indian Photographs, Columbia River Photographs, a Gustav Sohon Collection, and a Sheet Music Collection. The collections are very much oriented to the west side of the state (a search for "Spokane"  results in zero hits!) and are presented in ContentDM. But within these restrictions there tremendous things to be found at the WSHS site.

An Interesting Graduate Student Project

One of my graduate students, Shaun Reeser, is working to recover a long-lost government website for his MA public history project. Specifically, he will be working with some of the staff at the Washington State Archives, Digital Archives to recover the website of Ralph Munro, who served five terms as Washington Secretary of State from 1981 to 2001. Munro launched the first website for the Secretary of State's office in 1996, and the site was regularly updated until he left office in 2001. How to bring it back?

The Digital Archives has done something like this before, when we preserved the website of Washington Governor Gary Locke. In that 2005 effort the DA staff raced the clock to migrate Locke's website (an important public record) to the DA before it was taken down to make way for the website of Governor Gregoire. It was a pioneering project, but what Reeser will try to do is different. Munro's website was preserved in several versions at the Internet Archive Wayback Machine. But simply pulling the information off the Wayback Machine is not sufficient for establishing archival authenticity. And the versions of the website at the Wayback Machine are often incomplete, lacking some of the original images, for instance, and full of broken links.

Right now Reeser is trying to hunt down the original digital files and to interview Munro and members of his staff. He is also studying other efforts to spider and preserve government websites as historic documents. Does anyone know of similar effort that we should study?

[Image: Oh come on, you know.]

404, A Cautionary Tale

Don't ever create an extensive list of hyperlinks on your website. Just don't do it. It is so easy to get caught up in the excitement of cataloguing all the wonderful websites you find on your topic. Then you create your own website with a links page to these resources. And then--it falls apart. URLs change, websites go down, what was good becomes bad or redundant in the light of other new sites, and you have a mess on your hands.

Witness the unfortunate state of this University of Idaho site: Repositories of Primary Sources. It sounds so promising:

A listing of over 5000 websites describing holdings of manuscripts, archives, rare books, historical photographs, and other primary sources for the research scholar. All links have been tested for correctness and appropriateness.

Woohoo! A worldwide guide to all the websites for archives and special collections. This is exactly the kind of resource we need to keep track of all the other digital resources. So I eagerly navigated to the section for the Western United States and Canada and then to Washington and then to Central Washington University:

No problem. Lets try a different link. The East Benton County Historical Society? "Oops! This link appears to be broken." The Echoes of the Past Archive sounds interesting. "This domain is for sale. Please contact info@echoesarchive.com for more information." Nevermind, let's try my own Eastern Washington University--no, another 404 page. Of the first 15 links for Washington State, 11 are broken.

I do not mean to slam on the fine people at the University of Idaho, who obviously put a great deal of effort into creating this resource. No doubt they meant to maintain it, and no doubt more pressing matters have directed their attention elsewhere. This is just what happens to such endeavors.

So kids, never create extensive links pages. Or if you must, make them a wiki and leave a note asking users to fix anything they find that is broken.

Google's Book Search: A Disaster for Scholars?

Your humble Northwest History blogger is sometimes accused of being a Google fanboy. A fair cop. But you know who is not a Google fanboy? Geoffrey Nunberg, that is who. Over at the Chronicle of Higher Education Nunberg has a witty jerimiad, Google's Book Search: A Disaster for Scholars.

Nunberg's beef is with Google's sloppy and commercially driven metadata schemes. He demonstrates that even with such a basic item as date of publication, Google Books very frequently gets it wrong. This in turn often corrupts search results: "A search on 'Internet' in books published before 1950 produces 527 results; 'Medicare' for the same period gets almost 1,600." By comparing Google's data to that found in the catalogues of the contributing libraries Nunberg shows that these errors do in fact belong to Google, not to their partners.


Nunberg also whacks Google for the classification errors where books are placed in the wrong categories: " H.L. Mencken's The American Language is classified as Family & Relationships. A French edition of Hamlet and a Japanese edition of Madame Bovary are both classified as Antiques and Collectibles . . . An edition of Moby Dick is labeled Computers; The Cat Lover's Book of Fascinating Facts falls under Technology & Engineering."

Worst of all to Nunberg is Google's adoption of the Book Industry Standards and Communications categories for Google Books, which he describes as a modern commercial invention used to sell books, rather than a scholarly system of classification like the Library of Congress subject headings: "For example the BISAC Juvenile Nonfiction subject heading has almost 300 subheadings, like New Baby, Skateboarding, and Deer, Moose, and Caribou. By contrast the Poetry subject heading has just 20 subheadings. That means that Bambi and Bullwinkle get a full shelf to themselves, while Leopardi, Schiller, and Verlaine have to scrunch together in the single subheading reserved for Poetry/Continental European. In short, Google has taken a group of the world's great research collections and returned them in the form of a suburban-mall bookstore."

I think that Nunberg has a number of good points--point he gathers together to form a molehill, from which he conjures up a mountain. Google's metadata may be everything he says (and I think he is probably right) but how great a problem is that really? This scholar at least uses Google Books either 1) to locate a digital copy of a book I already know about, or 2) via a string of search terms. In the first case, it is not relevant to me that Google has classified Adventures of Huckleberry Finn under "wild plants" or whatever. I know perfectly well what it is, and just wanted to find a quote I remember.

In the second case, I might search for mentions of the Columbia River in books published before 1860. And suppose a faulty date in Google's database brings me to something written after 1860. So what? Surely when I click on the link and find myself reading Sherman Alexie instead of Lewis and Clark, I will notice the fact. (Actually I just did the search and on the first 10 pages of results I don't see any errors at all. Take that, Nunberg.)

So for which scholars exactly is Google Book Search a "disaster?" Nunberg cites "linguists and assorted wordinistas" who are "adrenalized" at the thought of data mining to "track the way happiness replaced felicity in the 17th century, quantify the rise and fall of propaganda or industrial democracy over the course of the 20th century, or pluck out all the Victorian novels that contain the phrase "gentle reader." But who does this? OK, I know that people do it, but most data mining of this type has always struck me as more of a parlour trick than actual scholarship.

The other thing Nunberg ignores is that metadata is not that hard to fix. Google already provides a "feedback" button on every virtual page so readers can report unreadable or missing pages. If we howl loud enough we could easily see similar feedback mechanisms on the "More book information" page so we could correct names and dates and categories.

Nunberg is absolutely correct to recognize the monumental importance to scholars of the Google Book Search project. It is vital that scholars take a critical stance that will push Google to improve the project and make it even more useful. His article is a valuable push in that direction.

UPDATE 9/3/09: Reader Ed points out that Geoff Nunberg also posted a nicely illustrated version of his article on the blog Language Log, and got a brief response in the comments from John Orwant, who manages the metadata at Google Books.

The Historical Treasure-Trove that is the UW Libraries Digital Collections

The University of Washington Libraries Digital Collections "features materials such as photographs, maps, newspapers, posters, reports and other media from the University of Washington Libraries, University of Washington Faculty and Departments, and organizations that have participated in partner projects with the UW Libraries. The collections emphasize rare and unique materials."

There are a lot of useful resources here--check out the Special Collections section for seventy-five wonderful digitized collections such as Alaska-Yukon-Pacific Exposition Photographs, Centralia Massacre and the Industrial Workers of the World Collection, 1912-1932, and the Vietnam War Era Ephemera Collection (from which the John Wayne poster featured here is taken.)

Good collections descriptions accompany all the collections and some have considerable additional material. American Indians of the Pacific Northwest, for example, has introductory essays, maps, and "bibliographies and links to related text and images as well as study questions that K-12 teachers may use as they develop curricula in their schools." They also have a blog to help users keep up-to-date with changes in their digital collections.

As wonderful as the site is, it could be more interactive. The images are difficult to download and save (which may be a feature rather than a bug!) and there is no Web 2.0 interaction. There is not even a way for users to flag an image that is obviously misidentified.

(Image: Cropped detail from John Wayne in No Substitute for Victory, created by the "Greater Seattle TRAIN Committee (To Restore American Independence Now)" and "Students for Victory in Viet Nam, Seattle Committee.")

Duke Digital Collections iPhone App

I am not sure if this is an oddity or a glimpse of the future, but Duke Digital Collections has developed what I believe is the first iPhone app for a digital archive. The app is really nicely designed and takes advantage of many of the iPhone's capabilities. Here is the demo they put up on YouTube:



And yet--I can't see using my iPhone to do historical research. What do you think, dear readers? Is this a very impressive novelty or something more?

[Hat tip to the frequently valuable Duke Digital Collections Blog--a nice example of an institutional blog.]