Pangur docsWork with corpora
A corpus is a set of sources that Analysis reads and explores together. This page shows how to create one, add and remove sources, give Pangur reading guidance, analyse the whole corpus, and export or delete it.
A corpus is a named set of sources that Analysis reads and explores together: a family's letters, say, or a run of court books, or one printer's output. It holds a name, a description, its sources and the reading guidance you give. The readings belong to the sources and the entities belong to the project, so a corpus is a way of looking at them rather than a container for them (the reason is given in How analysis works). Corpora belong to the project, which means every member of an organisation's project sees the same corpora.
Creating a corpus
You can start a corpus from several places, and each route ends on the new corpus's Sources view.
- From Analysis, choose New corpus. Give it a name (without one it is called "Untitled corpus"), search your library under Library, tick the sources and choose Create corpus.
- From a selection in your library, open Add to… and choose New corpus… under "A corpus to analyse", or the name of an existing corpus to add the selected items to it.
- From one item, choose Analyse… on its Analysis tab, then Select more sources to add others. Analyse just this item makes a corpus of that item alone, which is the route Analyse a source takes.
- From the catalogue, Analyse these… on a repository, collection or other level adds the manuscripts beneath it to a new or existing corpus. Only manuscripts linked to a library item can be added, and the page says how many of them are, for example "12 of 15 manuscripts here can be added to a corpus."
- From an ESTC collection, as described below.
Creating a corpus or adding sources to it reads nothing and costs nothing. Reading begins only when you choose to analyse.
If some sources could not be added, Pangur lists them with the reason. A source is refused when it is outside the project, or outside the collection the project is limited to.
Adding and removing sources
The corpus's Sources view lists each source with its status, which is one of "not analysed", "queued", "reading…", "analysed", "partly analysed" or "failed". A second badge shows the review state, "in review" or "reviewed", or "reviewed, since read again" when the source was read again after you reviewed it. A link such as "3 want a look" opens the review desk at the first mention that needs checking.
To add sources, choose Add sources, then Library or Catalogue. Under Library you search your library, tick the sources and add them. Under Catalogue you search by shelfmark or title, select manuscripts and choose Add with the count. A manuscript that cannot be added says why: it is not yet linked to a library item, its library item is no longer in your library, it is outside the collection this project is limited to, or it has no readable text yet. Choose Done adding when you have finished.
To take a source out, choose Remove from corpus beside it. Removing a source leaves its reading untouched.
A source can join a corpus before it has text or before it has been analysed. If a source is later deleted from your library, its row stays in the corpus marked as no longer in your library. A corpus with no sources left is deleted automatically the next time the list of corpora is loaded.
Telling Pangur what you know about the sources
Reading guidance is what you tell the model about a corpus's sources before it reads them. On the Sources view, find Reading guidance and choose Tell it what you know (or Change if guidance is already set). The form has these fields:
| Field | What to put in it |
|---|---|
| What these sources are | A short description, such as "Letters of the Keith family and their circle". |
| People who may occur, one to a line | Names and spellings the model should recognise. |
| Places that may occur, one to a line | Places, with the forms the sources use. |
| From the year / To the year | The period the sources cover. Units dated outside these years are flagged. |
Guidance is a hint about spelling and scope, not evidence: every quotation the model returns is still checked against the text. It applies the next time a source in the corpus is read and changes nothing that has already been read. When a source is read on its own, outside any corpus action, Pangur uses the guidance of the one corpus it belongs to that has guidance.
Assistants can also set a corpus's historical regions and period with the MCP tool update_analysis_context. The web form has no field for regions.
Analysing the corpus
Choose Analyse… on the corpus page. The Analyse this corpus dialog offers two choices. Only unanalysed reads the sources that have no reading yet. Read everything again reads every source again, except that sources you have marked as reviewed are left alone unless you also tick Also read again the sources you have reviewed. Corrections carry over.
The dialog lists each source with its model and estimated cost, or with the reason it will not be read ("already analysed", "already queued", "reviewed, left alone", or "cannot be read" when it has no text). A total line puts the costs together, for example "14 sources would be read, costing about $3.20." Choose Confirm and read.
Pangur queues one reading for each source, each read with this corpus's guidance, and the page says how many were queued. Each reading is charged to your daily model allowance. If the allowance runs out during a reading, Pangur keeps what it has read as a partial reading, and a reading that could not read anything at all fails with the allowance message. Read those sources again after the allowance resets at midnight UTC (Plans, limits and usage).
The corpus page
The corpus page's heading line counts what the corpus holds, for example "14 sources · 14 analysed · 412 entities · 3,190 mentions · 3 of 14 reviewed · 27 mentions want a look". Read opens the corpus's sources in the reader in order (The reading room), Analyse… is described above, and the Corpus menu holds the remaining actions: adding sources, renaming and describing the corpus, exporting it and deleting it.
The corpus's views (Index, Records, Timeline, the map and the rest) are described in Explore a corpus.
The corpus digest
A few minutes after a corpus changes, for example when a reading lands or a source is added, Pangur's server writes a corpus digest in the background. The digest is a model's reading of the whole corpus, its findings and every record, with quotations checked against the records. Ask reads the digest first when you ask about the corpus, and the themes shown under Concepts come from it. Because the digest is written for the project rather than for any one person, it is charged to the project owner's model allowance. If that allowance is spent, the digest waits for the next day.
Exporting a corpus
Export in the Corpus menu downloads the corpus's analysis. It appears once at least one source has been analysed, and the same menu is on each item's Analysis tab for a single source.
| Export | What it contains |
|---|---|
| Entities | A spreadsheet of the index. |
| Mentions | Every quotation, with unit, date and page. |
| Relations | What the sources state, with quotations. |
| Events | What happened, when and where. |
| Annotated text | Each unit's text with its marked spans, as JSON Lines. |
| Everything | One JSON file, for a program. |
Findings you rejected are left out, and spellings you merged into one entity are folded into the surviving entity. Each row records whether it came from the model or from a person. Viewers in an organisation's project can export.
Renaming or describing a corpus
Choose Rename and describe… from the Corpus menu, change the Corpus name or Description, and choose Save. Corpus names need not be unique.
Deleting a corpus
Choose Delete corpus from the Corpus menu and confirm. Pangur removes the corpus and keeps the analyses of its sources: readings, corrections and entities all stay, and the sources stay in your library and in any other corpus they belong to.
Importing an ESTC collection
The English Short Title Catalogue lets you export a collection you have made under My collections as a manifest file. Pangur can import that manifest as a library collection and a corpus together.
- Open Analysis and choose Import an ESTC collection.
- Choose the manifest file, which must be a
.jsonfile smaller than 16 MB with between 1 and 1,000 members. The import starts as soon as you choose it, and once Pangur has checked the manifest it tells you that you can leave the page. - When Pangur reports that the collection is in, choose Open the imported corpus.
Pangur's server imports the editions in the background, with their TEI texts where ESTC provides them. If the import fails, the page invites you to retry with the same manifest, and retrying with an unchanged manifest reuses the items and texts already imported. Importing reads nothing with a model. Assistants can do the same with the MCP tool import_estc_collection, and single ESTC editions are found as described in Find and add a work.
Analyse a source
This tutorial takes one primary source through its first analysis. You start the reading from the item, follow it as it runs, check how the source was divided, and read it again with a different division if you need to.
Review a reading
A reading made by a model is a draft until a person has checked it. This page shows how to work through a reading on the review desk, correct or add mentions, events and relations, decide which names are one person, and link entities to authorities.