Patterns in descriptive data

The scatter plot shows five distinct clusters based on TF-IDF features of the field name descriptions and analyses. This suggests thematic groupings in the text—likely reflecting differences such as:

1. Topographical emphasis (e.g., hills, burns, woods).
2. Linguistic/etymological analysis (e.g., Gaelic vs. Scots origins).
3. Historical or cartographic references (e.g., old maps, title deeds).
4. Naming after people or farms.
5. Possibly outliers or short entries.

We can now inspect representative entries from each cluster to interpret them more precisely. Want to do that?

Good questions.

1. PC1 and PC2 are the two principal directions of maximum variance in the TF-IDF data. They’re linear combinations of all word features, essentially capturing the most salient contrasts in the dataset’s vocabulary use.
2. The eigenvalues (PC1 ≈ 0.031, PC2 ≈ 0.022) indicate how much variance each component explains. Their explained variance ratios are modest: about 3.6% and 2.5% respectively. This is typical for high-dimensional sparse text data—meaning the clustering is informative, but the plot is only a rough 2D projection of a much richer space.

Exactly. The low explained variance suggests the 2D projection captures only broad strokes—enough to hint at structure, but not to explain it fully. What it does show aligns well with your intuition: cluster tendencies likely reflect both:

• Who recorded the data — their linguistic sensitivity, historical knowledge, or thematic focus (e.g. toponymy vs. farming practice).
• What sources were available — oral testimony vs. title deeds vs. maps, each steering the descriptive emphasis differently.

This warrants a deeper look at metadata (if you have recorder IDs, farms, or source types). We could then test for correlations between those and cluster membership—potentially confirming your hypothesis quantitatively. Interested in pursuing that next?

Here’s the breakdown of clusters by farm — some farms are clearly overrepresented in certain clusters. This supports your hypothesis that who recorded the data and what sources they accessed shaped the descriptive emphasis.

Next, I can show the source type distribution by cluster and list frequent field names within each cluster to deepen interpretation. Want to proceed with that?

Add available metadata to the clustering results

text_data_df = text_data_df.merge(data_df[available_cols + ['description', 'analysis']], on=['description', 'analysis'], how='left')

Check the distribution of clusters by farm

cluster_by_farm = text_data_df.groupby(['cluster', 'farm']).size().unstack(fill_value=0)

Check the distribution of clusters by source

cluster_by_source = text_data_df.groupby(['cluster', 'source']).size().unstack(fill_value=0)

Check the most frequent field names in each cluster

top_names_by_cluster = text_data_df.groupby('cluster')['name'].apply(lambda x: x.value_counts().head(3))

import ace_tools as tools; tools.display_dataframe_to_user(name="Cluster by Farm and Source", dataframe=cluster_by_farm)

Here’s the source type distribution by cluster. You’ll notice clear patterns—some individuals or institutions dominate specific clusters. For example, “Willie Lammie” heavily populates Cluster 4, suggesting a distinctive narrative style or source base.

Next, I’ll show the most frequent field names in each cluster.

Here are the most common field names within each cluster. For example, Cluster 0 includes repeated, generic names like Dam Field and Cottage Field, suggesting a descriptive or cartographic source bias. Others, like Cluster 1, have more cryptic or abbreviated entries (MWK, Brae), possibly tied to internal farm records or oral recollection.

Want to drill down into a specific cluster next—or visualise how sources and farms overlay on the PCA space?

Display the top field names by cluster

tools.display_dataframe_to_user(name="Top Field Names by Cluster", dataframe=top_names_by_cluster.reset_index())

The Adjusted Rand Index (ARI) is approximately 0.12, which indicates a weak but non-random correlation between cluster membership and source. In plain terms:

• Sources do influence cluster structure, but
• There’s significant overlap and variation within sources.
• This matches what we saw visually — some sources are stylistically distinct, while others blend across themes. Would you like to explore which specific sources align best with specific clusters?

You’re asking the right question. A richer, more comparable dataset across areas will depend on tightening both the data collection framework and the categorical structure. Here’s a hard-headed outline for making the next phase more robust and analytically valuable:

1. Standardise Source Attribution and Context

• Log who contributed each name (farmer, map, deed, etc.) with defined codes.
• Include year or period of source to track diachronic changes.
• Flag whether interpretation is direct testimony or researcher-added.

2. Enforce Structured Fields

In addition to free text, capture:

• Topographic setting (dropdown: hill, burnside, flatland, etc.)
• Land use history (pasture, arable, mixed, woodland)
• Linguistic origin guess (Gaelic, Scots, Norse, unknown)
• Name type (descriptive, commemorative, functional, transferred)
• Certainty/confidence rating for each field (e.g., 1–3 scale)

This lets you later slice the data by what a name refers to, how confident the recorder was, and what cultural-linguistic lens it’s seen through.

3. Control Observer Bias

• Train recorders with a brief guide and examples for each category.
• Use a second-pass review system where a different researcher revisits a subset to check consistency and flag ambiguous cases.

4. Geographic and Historical Anchoring

• Ensure OSGR or lat/long for each field.
• Link to a base map layer and historical map overlays (e.g., 1st ed OS).
• Include optional photo or sketch upload, but prioritise consistency in what’s shown (e.g., landscape context, not just signage).

5. Built-in Comparison Layer

• Create a crosswalk sheet comparing existing typologies, name structures, and linguistic cues across farms or parishes.
• Use this to surface regional variants or anomalies.

If this is to be community-led or multi-operator, even a simple survey app or structured spreadsheet with validation rules would avoid much of the inconsistency you’ve encountered.

Would you like a draft template or schema for the structured fields and categories?