Exercise V-A. The Proxy Integrity Assessment
Part V. Adversarial and Emerging Risks
Part V. Adversarial and Emerging Risks
Chapters 13 and 14 concern signals that mislead by construction or by intent. Chapter 13 specifies a Proxy Integrity Assessment in section 13.10 and testing beyond statistical fit in 13.11, and develops adversarial proxy manipulation in 13.13. Chapter 14 specifies an Adversarial GeoAI Exposure Assessment in section 14.9 and Adversarial Stress Testing in 14.10.
Fundamentals in play. These exercises use surrogate variables, correlation as evidence of substitutability, rank agreement between a proxy and its referent, and sensitivity of a ranking to injected records. The failure modes are that a proxy correlated with its referent in aggregate can diverge exactly where the decision is made, and that a ranking built from counts can be moved by a small number of fabricated records.
| At a glance | |
|---|---|
| Textbook sections | sections 13.2, 13.3, 13.4, 13.10, 13.11, 13.12 |
| Technical demand | Tier 1 and Tier 2 required. Python with pandas, numpy and matplotlib |
| Effort | two to three hours including the write up |
| Prerequisites | Exercises II-A and III-B |
Overview
Section 13.2 develops proxy logic, in which an observable or derivable measure represents a condition that is difficult or impossible to observe directly at the required scale, frequency or cost. Median household income routinely stands in for need, and it correlates with need well enough that the substitution passes casual inspection. This exercise measures where the substitution holds and where it fails.
Students compare a ranking of Mississippi tracts by median income against a ranking by the three variable need index, report the correlation between them, and then report the agreement among the top ranked units specifically. Section 13.11 argues for testing beyond statistical fit, and this exercise asks whether a strong global correlation guarantees agreement among exactly the units a program would fund. The program calls the three variable index the referent, and the index is itself a composite surrogate in the sense of Table 13.2.1, so the exercise measures how far two stand-ins for need disagree and cannot settle which one sits closer to need.
Step by step
- Establish the correlation. Tier 1. Change to the toolkit folder inside GeoAI_Exercises and type python exercise_5a.py. Record the correlation between the income proxy and the composite referent across all tracts, and state whether it appears strong enough to justify the substitution.
- Test where the decision happens. Tier 1. Record the Jaccard overlap between the fifty lowest income tracts and the fifty highest need tracts. Compare that overlap against the correlation from Step 1 and state which of the two a program manager should care about.
- Name the divergent units. Tier 1. Block 3 of the program prints the mean age, poverty, vehicle access and income of the tracts chosen by one ranking and not the other, and the review table exercise_5a_for_review.csv flags every tract's selection. Filter that table for ten tracts the index selects and income does not, and identify which index inputs set them apart.
- Map the disagreement. Tier 1. Record the map showing tracts selected by the proxy alone, by the referent alone, and by both. Describe the geographic pattern of each category.
Map, three categories. Tracts selected by the income proxy alone, by the composite referent alone, and by both.
- Screen for reliability. Tier 2. Section 13.11 allows a proxy that fails a test to continue under stronger measurement or a restricted operating domain. Block 4 restricts the comparison to tracts passing the reliability screen from Exercise III-B and reports whether agreement improves. Record the result, then change
CV_LIMITat the top of exercise_5a.py from 30 to 20 and to 40, rerunning after each change, and report how the overlap and the surviving tract count move. State what that says about whether the divergence is substantive or an artifact of measurement error.
- Apply the assessment. Tier 1. Section 13.10 organizes the Proxy Integrity Assessment around six dimensions, being Meaning, Mechanism, Transfer, Independence, Stability and Consequence. Rate the income proxy on each dimension, and cite for each rating a number from Steps 1 through 5 or state that no step measured it.
- Read the code and complete the handoff. Tier 1. Annotate the ranking comparison block and answer the decisions file.