How Pangur reads your sources

Pangur keeps two things about every work in a library. The record is the work's details: its title, creators, date, abstract and tags. The text is what the work itself says. The record exists as soon as you save a work, but the text has to be read out of the work's files, and Pangur's server does that in the background. Ask, analysis, the reading room and passage search all work from the text, while the library's own search box works from the record.

Extracting the textCopy Anchor Link

When a file reaches your library, Pangur's background worker extracts its text, divides it into passages, and stores the passages in an index that can be searched by words and, once the passages have been embedded, by meaning. The embeddings are made by a small model that runs on Pangur's own server, so no text leaves the server at this stage. None of this changes your files. The index is a separate store that Pangur builds from them.

The worker looks for files it has not read yet every 15 minutes, and the Mac and iPhone apps ask it to look straight away after an upload, so a new PDF is usually read within a few minutes. Reading also covers the short texts kept on items, which is to say notes, abstracts and highlights. When a file is replaced, Pangur reads it again, and when an item is deleted its text is removed from the index on the next pass.

Formats Pangur can readCopy Anchor Link

Pangur reads the text of these formats:

Some formats are stored and shown but not read for text, namely legacy Word .doc, legacy Excel .xls, and images. A PDF of more than 2,500 pages is not read either.

Recordings are a special case. Audio and video play in the reader, but their speech becomes text only when someone asks for a transcript, which uses the plan's transcription minutes (Record audio and video).

Scanned PDFsCopy Anchor Link

A scanned PDF has images of pages but no text layer, so there is nothing to extract. If the server has optical character recognition (OCR) switched on, the worker runs it on such a PDF as a last resort, up to 1,200 pages. OCR does not run on a PDF that already has a text layer, and it does not run on the files of manuscripts or letters, because OCR of handwriting produces confident nonsense. Text recognised by OCR is marked in the reading room as recognised from the scan, so that you expect misreadings.

A manuscript gets its text from a transcription instead. A transcription you save, whether you typed it or a model drafted it and you corrected it, becomes the manuscript's text and is indexed like any other (Transcribe a manuscript).

Which text an item hasCopy Anchor Link

An item can have several possible texts, and Pangur reads one of them at a time. It chooses in a fixed order: the texts of an arrangement, when an arrangement divides and orders the item's text, come first; then an imported edition, such as an ESTC TEI text; then a transcription; and finally the text extracted from a file, which is the reading copy's file or otherwise the largest file.

The Transcribe, Analysis and Outputs areas of an item say which text is in use, for instance that the item reads the text extracted from a particular file. The choice is not made separately on each screen. It follows the order above, so changing the reading copy, adding a transcription or importing an edition changes it everywhere. The reading copy itself is explained in Resources, copies and surrogates.

Checking whether a source has been readCopy Anchor Link

Pangur shows reading progress in a few places.

• When you upload a file onto an item, the upload control reports the outcome. It says that the file was saved and read, with the page count, or that only the first pages were read so far, or that the file is a scan with no text layer, or that its text could not be read, or that it is still being read.
• An item with no text says so, with the likely reason: a file added just now may still be being read, the pages may need transcribing, or there may be no file to read.
• In the reader, Extracted text shows the text Pangur holds for a file, and tells you when it holds none.
• On Ask, while nothing in the library has been read yet, a notice says "Your library is being read." with a count of the sources read so far and of those that cannot be read. Until reading finishes, answers come from titles, abstracts and notes. A library that is only catching up with a few new files shows no notice.

Reading the index is not background work that you started, so it does not appear in Activity.

Library search and passage searchCopy Anchor Link

The search box in the library matches the record: title, creators, year, DOI and ISBN, tags, publication and abstract. It does not look inside files. To find words inside your sources, ask a question in Ask, which shows the matching passages as its sources (and only the passages, when no model is switched on), or use Find in text in the reader or the reading room. The library's own search is described in Search and filter your library.

The library mapCopy Anchor Link

The library map is a short description of each work, written once from the work's whole text and stored. An entry records what the work argues, what it covers, how it is structured, and where it is thin. Ask reads the map before every question, so that it knows what each work is about before it searches passages. The map is also how Ask can tell a work that is absent from the library from one whose passages simply did not match.

Pangur writes a work's map entry with a model after the work's text has been read, and writes it again when the text changes. The model use is charged to the daily model allowance of the project's owner, and when that allowance is spent the remaining entries wait for the next day. The map is not written at all when no model is switched on for the deployment. You do not edit or view the map directly. Ask uses it, and assistants can read it through the library_map tool (Models, assistants and who pays).

Works with no full textCopy Anchor Link

A work has no full text when Pangur holds its record but no readable text, because it has no file, its file is a format Pangur does not read, or its scan could not be recognised. You can still find such a work by its record, cite it and organise it, but nothing inside it can be searched or quoted.

Ask treats such a work cautiously. Asked about it, Ask replies that the work is in your library but that no full text is held, declines to answer about the book from the record and abstract alone, and offers to look for a PDF or asks you to add one. Analysis refuses to read an item without text, and its message says to transcribe the pages or add a file with text, noting that a file added a moment ago may still be being read. To give a work a text, add a PDF or another readable file, link or upload page images and transcribe them, or import an edition. How Ask uses the text it does have is described in How Ask answers.