A bank tests its AVM by comparing the model's estimates with the prices of homes that sold last quarter in its trade area. The vendor's model had already incorporated those sales before the test was run. What is the main weakness of this validation?
Correct Answer
A) The model already had the sales, so the test can overstate its accuracy
To ensure unbiased results, the guidelines call for comparing AVM results with actual sales in a specified trade area or market before that information is available to the model. A model that already contains the test sales can appear more accurate than it is. Actual sales in a trade area are the right benchmark, and banks should not rely only on vendor validation.
Why This Is the Correct Answer
Option A is correct because the guidelines call for comparing AVM results with actual sales before that information is available to the model, to ensure unbiased test results. When the model has already incorporated the sales, its estimates may simply echo the prices it learned from. The test can therefore overstate the model's accuracy on future loans.
Why the Other Options Are Wrong
Option B: The bank should rely on the vendor's validation instead of its own test
Option B says the bank should rely on the vendor's validation instead of its own test. The guidelines state that an institution should not rely only on validation representations provided by an AVM vendor.
Option C: Actual sales prices are not a suitable benchmark for testing an AVM
Option C says actual sales prices are not a suitable benchmark. The guidelines specifically call for comparing AVM results with actual sales data; the problem is when those sales became available to the model.
Option D: The test should cover a national sample rather than one trade area
Option D says the test should use a national sample rather than one trade area. The guidelines call for testing against sales in a specified trade area or market, and the bank's level of geographic analysis is something its procedures should specify.
Test on sales unseen
A fair AVM test uses sales the model did not know about when it produced its estimates.
How to use: Use this phrase whenever a validation scenario uses sales data that may already be inside the model.
Exam Tip
When a validation question gives an otherwise sound design, look for the timing of the test data relative to the model's data.
Common Mistakes to Avoid
- -Using sales the model has already ingested to test it
- -Relying on vendor accuracy statistics without independent testing
Concept Deep Dive
Analysis
The question asks the candidate to spot a flaw in a validation design. Comparing against actual sales in a trade area is correct, so the flaw must be the timing: the model had already absorbed the test sales. A model tested on data it has seen can look more accurate than it will be on new transactions.
Background Knowledge
AVM validation compares model estimates with independent evidence of value, usually subsequent sale prices, to measure accuracy and bias. Independence of the test data from the model's inputs is central to obtaining a fair measure of performance.
Real-World Application
A bank records AVM estimates on pending purchase loans before closing and later compares them with the recorded sale prices. Because the estimates were captured before the sales were public, the comparison gives an honest measure of model accuracy.
More Emerging Methods Questions
A loan is eligible for Fannie Mae's value acceptance + property data option, and the lender elects that option. What does the lender obtain in place of an appraisal?
An appraiser uses a building-permit file to investigate an addition. What is an important limitation of that source?
A real estate salesperson, acting solely in a brokerage role, prepares a CMA to help a listing client choose an asking price. Which statement is best?
An appropriately credentialed appraiser is offered a desktop assignment on an unfamiliar property type. A necessary competency gap is known before agreement. If the appraiser elects to proceed, what is required?
How should an appraiser identify potentially distressed transactions in a data set?
Why should an appraiser identify the data sources used in the report?
How should an appraiser treat a value estimate found on a consumer real estate website?
An appraiser builds a spreadsheet regression to support adjustments. How is it treated?
May an appraiser use data from a prior assignment for a different client?
The intended lender’s applicable assignment conditions require an interior inspection. The client asks the appraiser to deliver a desktop appraisal instead, and the inspection condition has not been authorized for modification. What should the appraiser do?
People Also Study
Real Estate Market
13.6% of exam
Property Description
11.8% of exam
Land or Site Valuation
4.5% of exam
Sales Comparison Approach
16.4% of exam
Cost Approach
13.6% of exam
Previous Question
Why should an appraiser investigate measurement-method differences between subject and comparable square-footage data?
Next Question
A bank uses an analytical tool that reports a price trend for the subject’s ZIP code but no estimate of the property’s market value. Under the Interagency Guidelines, can this output serve as an evaluation?