Norma vigente
CONTENTS
1. Subject matter, scope and authority
2. Principles for the use of AVMs by appraisal companies
Only AVMs that follow generally accepted valuation practices for the valuation of properties that have a certain degree of homogeneity should be used
2.2 The AVMs should include information relating to both the market value and the mortgage value
2.3 The methodology used should be sound, and regularly and appropriately tested, and should ensure that the property valuations made using AVMs are traceable
2.4 The AVMs used should be specified and documented
2.5 The AVMs used should be calibrated and the process documented
2.6 The sufficiency and quality of the information available should be checked to ensure that a commission for a mass AVM valuation can be performed
2.7 A sufficient sample of full individual appraisals should be taken from the properties included in a mass valuation in order to backtest the results obtained using AVMs
2.8 All the information relating to the property valuations made using AVMs should be made available to the Banco de España
ANNEXES
ANNEX 1 – Example of how to calculate the number of properties in the sample used to check the values
ANNEX 2 – AVM valuation report
ANNEX 3 – AVM datasheet
ANNEX 4 - Ratios to measure the valuation reliability
Automated Valuation Models (AVMs) are property valuation methods that use statistical models developed by means of computer programs on the basis of large databases. They enable mass valuations to be performed at low cost, taking into account the specific characteristics of each asset valued, while assuming a certain degree of error in the results, since the model, by definition, does not include all the variables that hypothetically may affect value. Without prejudice to their statistical base, throughout the process of obtaining a valuation, AVMs incorporate the expert judgement of the appraisal company, both in the construction of the model and its specification and testing, and in the selection of the appropriate model for each specific valuation commission.
Appraisal companies use AVMs, inter alia, to carry out commissions received from credit institutions for use in the accounting valuation of properties that make up collateral and foreclosed assets, in accordance with Annex 9 of Circular 4/2017 of 27 November 2017 on public and confidential financial reporting standards and formats (“Annex 9”).
These guidelines are for use by appraisal companies registered with the Banco de España and are applicable to the work mentioned in the previous paragraph. The guidelines identify the recommended best practices for the definition and application of sound valuation procedures using AVMs.
These guidelines are published under the Banco de España’s powers to supervise appraisal companies, as laid down in Article 15(1) of Royal Decree 775/1997 of 30 May 1997 on the legal regime for the approval of appraisal companies and services.
Under Annex 9, the valuation of property collateral and foreclosed properties shall be carried out in accordance with sound procedures undertaken by appraisal companies. Specifically, the Banco de España expects the following principles to be observed in order to achieve this objective:
2.1 Only AVMs that follow generally accepted valuation practices for the valuation of properties that have a certain degree of homogeneity should be used
AVMs are only suitable for valuing properties that have a certain degree of homogeneity, i.e. with characteristics that can be replicated (point 78.b of Annex 9), and that are located in an active market, so that appraisal companies have sufficient information to model and process them en masse.
AVMs must follow generally accepted valuation practices (point 78.b.ii of Annex 9). Examples of such models are: a) hedonic multiple regression models; b) comparables models, and c) mixed models that combine the first two types of model. Under no circumstances is it considered good practice merely to adjust a previous valuation in line with a general price index. That said, valuations may be supplemented by an analysis of price series based on a large number of actual transactions or full appraisals of properties with similar characteristics, provided that such series have sufficient data to establish adequate territorial granularity in the definition of local markets (at least by postcode) and that the valuations to which they are applied are full appraisals that have been carried out by the same company that compiles the series within the last three years.
a) Hedonic multiple regression models use multivariate statistical techniques (basically multi-linear regression) to estimate the value of a property based on the key features affecting its value, including, when considered necessary, socio-demographic elements of the area where the property is located. Accordingly, each of the properties to be appraised, and the specific features of its location, must be adequately characterised. These models may be strictly statistical (such as hedonic regression models), adaptive or based on artificial intelligence technology (such as neural network models).
The type of calibration will depend in turn on which model is used. In all cases, the data the model works with must be complete and accurate, and the samples must be representative.
b) The comparables model is based on the selection of a sample of properties located in the same area where the asset to be valued is located, with similarities in terms of construction, surroundings and sale opportunities, for the purpose of establishing a relationship between the properties in the sample and the asset to be valued.
c) Mixed models are a combination of the first two methodologies, albeit with an added subjective component regarding the assignment of values on the basis of the different models.
The values obtained by means of AVMs may be modified by expert judgment or by additional checks performed by an appraiser. In such cases, they are called appraiser-assisted AVMs (AVM(A)s). These models tend to be more accurate than AVMs, but they are also more costly in terms of time and price.
2.2 The AVMs should include information relating to both the market value and the mortgage value
Although they are concepts applicable, in principle, to full appraisals, owing to their well-established use valuations performed by AVMs are expected to yield two technical values for each property: the market value and the mortgage value, as defined in Article 4 of Order ECO/805/2003[1], since the market value should be used as the reference value in valuations of foreclosed properties (point 166 of Annex 9), and the mortgage value as the reference value when assessing real estate collateral for mortgage lending transactions (point 78.a.i of Annex 9).
The market value is expected to envisage a reasonable average sale period and to be in line with the experience of sales existing in each market.
2.3 The methodology used should be sound, and regularly and appropriately tested, and should ensure that the property valuations made using AVMs are traceable
In accordance with Article 3(1)(e) of Royal Decree 775/1997, appraisal companies shall have a sound and regularly and appropriately tested methodology enabling a rigorous estimation of the current prices of properties in their respective local markets.
To confirm the rigour of the estimation, the results of the model are cross-checked with real and current transactions at market prices or with full individual appraisals performed, in both cases, within the 12 months preceding the issuance of the valuations made using AVMs by the appraisal company for comparable assets, in terms of location, size, construction or any other relevant circumstance for comparison purposes.
Also, an appropriate methodology will ensure the traceability of the valuations obtained by means of AVMs for the properties and the exclusion of properties that cannot be valued using AVMs because they do not meet the minimum requirements relating to quality and sufficiency of information (see Section 2.6), or because they are unique or not suitable for valuation en masse (see Section 2.1).
2.4 The AVMs used should be specified and documented
AVM specification is the iterative process of calibration and adjustment through which the appraisal company develops the structure of its model until the definitive version is obtained.
Prior to the specification of the AVMs, the appraisal company should determine their scope of application and define the type of property and the areas or sub-markets in which the models are to be applied, in accordance with the relevant calibration tests.
An appropriate way of recording this process is for the detailed information of the model specification to be set out in a duly updated document that includes the following:
- A detailed list of data sources on the basis of which the model was constructed: appraisals, offers from real estate portals or real estate agents, and cadastral, registrar and notary values, among others.
- A list of the different types of properties considered and of the data used for each property type.
- The variables used in the model on the basis of the data available for each type of property, area or market considered.
- Work performed to edit the data beforehand: criteria for dealing with errors, information shortcomings and extreme values in the property data. Appraisal companies should also establish criteria relating to the treatment of extreme values (outliers) and the techniques for processing such values and errors and information shortcomings (winsorising or trimming).
2.5 The AVMs used should be calibrated and the process documented
AVM calibration refers to checking the model’s specific structure, including its adjustment function and coefficients, through the model’s regressors generated by its database. Model calibration may be purely statistical or adaptive.
The information on calibration of the model is expected to be recorded together with that relating to its specification, including, at least, complete and accurate data, as well as the representative samples on which appraisal companies base their calibrations. The information should also include, inter alia, the number of observations applied, starting ranges (minimum and maximum) for each coefficient and the final value considered.
It is important to note the following aspects relating to the different calibration processes and to their documentation, for consideration by appraisal companies, using measures of reliability compatible with the valuation method being calibrated:
- Purely statistical calibrations should meet certain minimum requirements, such as the normal distribution of errors, the non-correlation of variables, homoscedasticity of errors and linearity. It is recommended that appraisal companies verify compliance with these requirements and report the results of the calibration tests at a confidence level of no less than 95%.
Also, it is considered good practice for appraisal companies to use control samples for the purpose of checking whether the results are consistent and to recalibrate the model if they are not.
In addition, to guarantee the quality of the model, it is advisable for the level of significance of the coefficients of each of the variables to be assessed, so that only parameters with a specific level of significance (at least 95%) and with consistent effects are incorporated.
It is also advisable for the goodness of fit be measured. For this purpose, indicators such as the coefficient of determination R2 or the Akaike maximum likelihood comparison measures (AIC) may be used.
- In adaptive calibrations the pre-calibration of starting coefficients is equally important in order for them to be at appropriate levels. In calibrations of this type goodness of fit will be determined by the Pseudo-R2.
Appraisal companies are expected to submit information on the performance of their models to the Banco de España.
2.6 The sufficiency and quality of the information available should be checked to ensure that a commission for a mass AVM valuation can be performed
Prior to accepting a commission to perform a valuation using AVMs, appraisal companies are expected to monitor:
a) The sufficiency of the data available on the properties’ characteristics, such as cadastral references, postcode, built and useful areas, area of balconies/patios, number of bedrooms, garages and adjoining storage rooms, age and state of repair, year of last known full refurbishment, status as government-sponsored or open-market housing and, where possible, the assets’ geopositioning coordinates (latitude and longitude).
b) The quality of the available data on the properties: appraisal companies are expected to only accept commissions for the issuance of valuations using AVMs when the commission document contains a general mention of the database quality review by the internal audit department of the credit institution, in accordance with point 78.b.iii of Annex 9 of Circular 4/2017.
2.7 A sufficient sample of full individual appraisals should be taken from the properties included in a mass valuation in order to backtest the results obtained using AVMs
Best practice involves appraisal companies sampling a sufficient number of full individual appraisals in order to check the precision of the appraisals obtained using AVMs (point 78.b.iii of Annex 9). The sample of properties should be different to those samples used for the specification and calibration of the model and the date that the appraisals are conducted should be close to the AVM valuation date.
The result of this backtesting should be stated in the report which, along with the valuations, the appraisal company may provide to the credit institution as a result of its work.
The Banco de España expects the details of these tests to be properly documented, so that they can be assessed by a third party, which can draw conclusions as to the suitability of the results obtained by AVM. This documentation should also contain an analysis of the differences between the errors in the training and backtesting phases
In order to determine the sample size of the properties to be valued individually, they should be stratified by area or sub-market and by type so that the minimum sample size, with an acceptable minimum reliability level and a specific maximum level of relative estimation error, is set for each group.
Annex 4 of these guidelines develops the ratios for measuring the reliability of AVM valuations and offers guidance regarding what levels are reasonable, in accordance with generally accepted practice.
The sample size (n) should be determined by the following parameters:
or:
A 95% confidence level and a 5% relative estimation error are considered suitable values, although depending on the number of properties valued by AVMs and the effect the model errors may have on the value of the assets, these values can be modified. Confidence levels of less than 90% and relative errors greater than 10% are not considered suitable under any circumstances (an example is provided in Annex 1). Also, in order to adjust the precision criteria to the level of risk (as point 84 of Annex 9 does, by requiring more frequent valuation in higher risk situations) in commissions from credit institutions, which specify, separately, that the properties to be valued are foreclosed or collateral for mortgage loans that are non-performing or under special monitoring with LTV ratios of over 70 %, the Banco de España considers it appropriate to establish as minimum requirements a 95% confidence level and a 5% relative estimation error.
2.8 All the information relating to the property valuations made using AVMs should be made available to the Banco de España
To enable it to carry out the supervisory function specified in Article 15(1)(b) of Royal Decree 775/1997, appraisal companies should make available to the Banco de España an internal document setting out all the information relating to the AVMs used, including the procedures and tests performed to calibrate and check them, which should be checked against the principles of these guidelines. If more than one methodology is used, the appraisal companies should present the information on each one separately. The available information should be sufficient to be able to assess each commission received by appraisal companies.
Annex 2 contains an example of an AVM valuation report and Annex 3 includes a summary datasheet for each AVM used.
By way of example, the table below shows the estimated sample of a population of 10,000 properties for a confidence level of 95%, depending on the COV and the maximum relative error allowed.
For example, for a COV of 60% and an error of 10 %, a sample of 136 properties is required (see shaded cell).
The level of the model estimation error, the relative errors for the checks performed and other parameters deemed appropriate should be determined on the basis of the results of such tests. The confidence levels for each estimate should also be indicated.
This annex sets out what the Banco de España considers an AVM valuation report should contain.
An AVM valuation report should be a formal document, in paper or electronic format, with the letterhead of the issuing appraisal company.
The report should include the name and registration number of the issuing company in the Register of the Banco de España and the number of pages it consists of, and it should be signed by the appraisal company’s representative and by the professionals who drafted and supervised the report. In the case of an AVM(A), the name of the professional who has intervened in the final values should be included in the report. Signatures may be handwritten or electronic and the appraisal company should set up procedures ensuring that the documents signed are unalterable.
The report should contain sufficient information on the commissioning entity, the properties valued and the model used. By way of illustration, the report may contain the following sections (from A to L), to be completed taking into account the principle of proportionality:
A Entity commissioning the automated valuation, purpose of valuation and observation of the principles of the guidelines
- Name and Banco de España registration code of the credit institution which has commissioned the automated valuation.
- Purpose(s) for which the AVM valuation is commissioned.
- Indication as to whether the performance of the AVM valuation has observed the principles of these guidelines.
B Characterisation of properties to be valued
- Number of properties whose valuation is commissioned, grouped by type (e.g. flats, terraced and detached houses, offices, commercial premises, etc.) and by location (at least according to postcode).
- Summary of average values of main variables, such as surface area and age.
C Scope of application of model
- Identification of real estate items to which AVMs are to be applied.
D Illustrative identifying characteristics for estimating the property values
- Identification of property: country, province, municipality, police address, cadastral reference, standard geographic coordinates (WGS 84).
- Property size: adopted area, land area, if applicable.
- Normal use of property: residential, commercial, offices, parking, etc.
- Type of property: flats, terraced and detached houses, etc.
- Age and state of repair; if the property has been reformed, percentage reformed and year of reform.
- Occupation status of property: owned, leased or empty.
- Characteristics of property: floor number, orientation, views, lighting and sunlight, number of rooms, bedroom/bathroom ratio, heating and cooling systems, natural gas, quality of finishes and equipment, etc.
- Other elements included in the property reference (if appropriate): storerooms, parking spaces, etc.
- Characteristics of the building where the property is located: existence or lack of homeowner association membership; existence of common or private facilities, such as elevators, sports facilities, pools or recreational areas; number of floors in the building above and below ground level, height of building, etc.
- Property location: degree to which it is in an established area, as well as communications and public and private facilities; socio-economic status, average age of properties and supply-demand ratio in the area; peripheral or central location within the geographic area examined and access to main road network.
- Analysis of supply and demand: demographics and capacity for creating and absorbing supply in the surroundings. Estimated time to sell the property in the valuation.
E Descriptive analysis of the portfolio analysed
- Summary by type and geographic location of the properties that it will be possible to value since the minimum information on the necessary variables established to perform a valuation of each of them is available.
- Percentage of properties it has not been possible to value using an AVM because the minimum requirements of information on the variables considered are not met.
- Properties which cannot be valued because they are considered unique or they cannot be replicated or on account of their location. [2]
F Type and description of models applied and software used
- Model types applied to each population segment analysed and explanation of the methodology used.
- Latest model calibration date, database on which it was based and software used.
G Model testing and calibration
- Model testing and calibration carried out by the appraisal company, reporting the model’s absolute and relative errors.
- Statistics of the coefficients that make up the model, level of compliance with error homoscedasticity levels, normality, independence and representativeness of the data on the basis of which the model was constructed.
H Value data and calculation
- The values obtained by the AVM for each property (market value and mortgage value) and the confidence interval limit of 95%, indicating the model used to obtain the valuation and, in the case of an AVM(A), the professional involved in the decision to establish the value.
- If several models have been applied, the value obtained for each property by each model should be indicated.
- Summary information on the result, by type of property and location (at least by municipality), and in electronic format, detailing the values for each of the properties appraised, as indicated in point L on the documentation annexed to the report.
I Backtesting
- List of checks performed of results obtained by the model to verify their suitability: random performance of individual appraisals or comparison of values with sales completed close to the valuation date.
For example, the number of full individual appraisals should be sufficient to reach a confidence level of 95%, for a sampling error lower than 5%, in accordance with the following formula:
where there is a normality assumption for the errors, �� is the total number of properties valued by AVMs and �� is the standard deviation of the value of the properties estimated using AVMs.
Calibration adjustments to the results obtained, if appropriate, indicating the confidence level of conclusions.
J Model backtesting ratios
Table of ratios obtained from the backtesting of the model to check its quality (measurement of the precision and bias of the valuations):
K Date of issue and signatures
- Date of issue of the report.
- Name, signatures and position of persons signing the report, and name and position of all other specialised technicians involved in drafting the report.
L Documentation annexed to the report
- Detail, in a computerised medium, of all the properties analysed and the values recorded for each one (market and mortgage values), indicating in each case, at least, the characteristics required which enabled the valuation to be performed using an AVM.
- Properties which could not be valued in an automated manner.
This annex sets out what the Banco de España considers should be included in an AVM datasheet.
In international practice a series of ratios is used to measure the reliability of the results of AVM appraisals. These ratios are calculated by comparing the values estimated by AVMs with real transaction prices, or failing that, with the prices obtained in individual full appraisals.
Two types of ratios are used to check the quality of the model, precision ratios (MAPE, RMSE, COV, COD, COC, FSD, STD and HIT ranges) and ratios indicating bias (mean ratio, median ratio and PRD ratio).
The MAPE (mean absolute percentage error) and RMSE (root mean squared error) ratios measure the model’s error level. The MAPE ratios measure the mean absolute error between the AVM values and the comparison values[3]. Being a percentage measure it is comparable across different AVMs. Absolute error levels of up to 10% are considered normal, but over 13% is considered higher than reasonable. RMSE ratios measure the error in absolute terms and are therefore not appropriate for comparisons. However, they do allow us to obtain an idea of the magnitude of the error.
The COV (coefficient of variation) is the ratio of the standard deviation to the mean of the differences, and is a dimensionless measure of dispersion. The COD (coefficient of dispersion) is the average of the differences in absolute values divided by the median ratio and measures the variability or uniformity of the appraisals. It is considered more appropriate than the COV, although both ratios are dimensionless and allow different AVMs to be compared.
The COC (coefficient of concentration) measures the percentage of AV/SP[4] ratios that fall within a given percentage of the median, thus constituting a measure of uniformity. It is usually established that if 50% of the ratios are within ± 10% of the median, the COC is 50. The higher this value, the better the AVM valuation.
The FSD (forecast standard deviation) ratio measures the standard deviation of the estimation error percentages. The percentage error is defined as:
where the comparison value may be either the full individual appraisal value or the sale transaction value. This ratio gives an idea of the magnitude of the bias in the valuation.
The STD ratio is the standard deviation of the differences in the appraisals. This is a measure of the appraisals’ dispersion, in unit terms, and is therefore comparable to other AVMs.
The HIT ranges test indicates the percentage of valuation differences that fall within a given interval. It is considered adequate if at least 50% of the differences between valuations fall within an interval of ±10%.
As regards the ratios measuring the AVM's bias, the mean ratio and the median ratio are measures of the position of the valuations: they indicate the sign and magnitude of the bias, depending on whether the AVM valuations are above or below the comparison values. The median ratio is more robust than the mean ratio. By way of guidance, the values considered suitable for both ratios are between 0.9 and 1.1. Values below 0.85 and over 1.15 are not considered suitable. The ratios are expressed as a decimal fraction, making it possible to compare them.
And, lastly, the PRD ratio (price-related differential) measures the vertical biases in the valuation, i.e. whether the bias increases or decreases as the value of the variable analysed increases. It is calculated as:
