Recommendation E.421 SERVICE QUALITY OBSERVATIONS ON A STATISTICAL BASIS 1 Definitions 1.1 service observation F: observation de la qualit‚ de service S: observaci¢n de la calidad del servicio Monitoring to obtain a complete or partial assessment of the quality of telephone calls, excluding test calls. 1.2 manual observation F: observation manuelle S: observaci¢n manual Monitoring of telephone calls by an observer without using any automatic data-recording machine. 1.3 automatic observation F: observation automatique S: observaci¢n autom tica Monitoring of telephone calls without an observer. 1.4 semi-automatic observation F: observation semi-automatique S: observaci¢n semiautom tica Monitoring of telephone calls using equipment which records some data automatically. For example, equipment in which information, such as exchange being observed, number dialled by the subscriber, metering pulses and time of call, is recorded automatically on some means suitable for data processing. The observer merely has to key in a code indicating the condition observed. 2 Relative merits of manual, automatic and semi-automatic observations 2.1 The three methods mentioned above in SS 1.2, 1.3 and 1.4 are not exclusive; for example, automatic observations may be used to supplement observations taken by an operator. It was considered in 1968 that the need for automatic observations would increase in view of the heavy cost associated with manual or semiautomatic observations on the rapidly expanding international network. It was also considered that automatic observations would not entirely supersede observations taken by an observer within the foreseeable future. The relative merits of the three methods can be assessed as follows: 2.2 Manual observation Provides all the data required in Tables 1/E.422 and 1/E.423. Observations can be carried out with a minimum of equipment. Observations can permit the detection of a number of abnormalities which cannot be detected automatically, e.g. very poor speech transmission (item 5.2 of Table 1/E.422), or difficulty with audible tones encountered in the international service (item 6.4 of Table 1/E.422). 2.3 Semi-automatic observation Provides all the data required in Tables 1/E.422 and 1/E.423. There is a saving in staffing costs compared with manual observation. Greater accuracy compared with manual observation is possible due to the fact that there is an automatic recording of the number dialled, the time of the call, etc. It is possible for the observer to devote greater attention to the more critical conditions being checked during observations of calls. The results are produced in a form suitable for subsequent mechanized analysis. Owing to the reduction of costs it is possible to obtain a larger sample for the same expenditure. Semi-automatic equipment may be converted, during certain hours of the day, to automatic operation. 2.4 Automatic observation Operating cost is minimum (staff reduction). Continuous observation is possible. It is possible to have a larger sample. Human error is eliminated. Automatic processing of data is facilitated. Conversational privacy is ensured. Control of the time at which observations are made is facilitated. Some of the differences between internal and external automatic observations are given below: Fascicle II.3 - Rec. E.421 PAGE1 2.4.1 I automatic observations can be made in the switching centre itself, on the incoming side or the outgoing side or in between, according to the way the switching centre is engineered: a) Only line signals, such as seizure, answer, etc. can be monitored, and also register signals as long as they do not pass through the exchange in an end-to-end signalling procedure. b) Signals received are only monitored if the exchange itself operates correctly in that respect. c) Item b) applies also to outgoing signals. If there is a fault in the exchange it can happen that signals have not been sent in the appropriate way without the exchange being aware of it. More information on this type of observation technique is given in Recommendation E.425. 2.4.2 E automatic observations are made by means of monitoring equipment which is supervising the traffic on incoming or outgoing lines: - All signalling signals can be monitored. - The detection of tones, speech and data is possible if advanced equipment is used. - This observation technique provides all the data required in Table 2/E.422 and Table 2/E.423. - The application is very flexible and can be used instead of manual or semi-automatic observation techniques. 3 Time of observations The results of all observations taken over the whole day should be recorded in Table 1/E.422 or Table 2/E.422. In the case where observations are not taken over the whole day the observation period is recorded under the heading "Time of observations" and should include the three busiest hours of the day. 4 Observation access points 4.1 Observations for Table 1/E.422 or Table 2/E.422 should be carried out from access points as close as possible to the outgoing international exchange. The following access points can be considered: For definitions of test access points see Recommendation M.700. See also Recommendation M.110. i) outgoing set of an international circuit ("exchange" side), i.e. international circuit access point1); ii) incoming relay set of a national circuit; iii) link circuits of the international exchange. Observations will be made only while the call is being set up, and a few seconds after the called subscriber's reply. When the circuit access point is used for observation of international calls it is possible that the service quality of the international exchange may not be checked by either international or national observation programmes. Preferably, and where technically feasible for the most complete results, observations for Table 1/E.422 should be carried out as close as possible to the international exchange on the national side. This would be more representative of service to the subscriber, and allows observation of call failure at the outgoing international exchange. Where it is not possible to make the distinction between failures in the outgoing international exchange, and failures beyond this exchange, or where there is a meaningful advantage in doing so, observations should be taken on the outgoing side. It is necessary to state in Table 1/E.422 or Table 2/E.422 the access point where the observations have been made, as observations obtained at each one of the three access points mentioned above are not comparable. 4.2 Observations for Table 1/E.423 must be carried out from access points on the operators' positions. 5 Number of observations 5.1 Service observing programmes should be established in such a manner that statistical results obtained be as reliable as practicable bearing in mind the cost of obtaining large samples. 1) For definitions of test access points see Recommendation M.700. See also Recommendation M.110. PAGE4 Fascicle II.3 - Rec. E.421 5.2 According to the studies carried out by the CCITT in 1964-1968, the quantities shown below are considered the minimum quantities to provide a general indication of the quality of service. 5.2.1 Table 1/E.422 The minimum number of observations per outgoing circuit group for Table 1/E.422 should be 200 per month when more than 20 circuits are included in a group, 200 per quarter when there are between 10 and 20 circuits in a group and 200 per year if there are less than 10 circuits in a group. 5.2.2 Table 1/E.423 The minimum number of observations for Table 1/E.423 should be 200 per quarter when there are more than 20 circuits in the group, 200 per semester when there are between 10 and 20 circuits and 200 per year when there are less than 10 circuits in the group. 5.2.3 Transit traffic Where an outgoin t group also carries transit traffic it is desirable to obtain data for each destination country reached via this circuit group. In principle, the number of observations for each destination should be obtained as indicated above. To accomplish this, one should use for each destination country its corresponding number of erlangs and derive from these erlangs a theoretical number of circuits. However, where only a very small amount of traffic is handled, e.g. less than 5 erlangs, each Administration may wish either to make a smaller number of observations or (e.g. in case of no complaints) no observations at all and rely on the information obtained at the transit exchange. 5.3 The number of observations specified above will provide a general indication of results on quality of service in certain broad categories. Administrations may desire more accurate results especially for the individual categories in Table 1/E.422. Attention is drawn to Table 1/E.421 which gives the number of observations required to obtain a certain degree of accuracy. TABLE 1/E.421 Expected percentage Number of observations of a random sample required to predict rate with 95% confidence the true percentage of failure with an accuracy of: of failure ± 25% ± 30% ± 35% ± 40% ± 45% ± 50% 2 3136 2178 1600 1225 1030 880 4 1536 1067 784 600 500 440 6 1003 696 512 392 Fascicle II.3 - Rec. E.421 PAGE1 330 290 8 736 511 376 288 245 215 10 576 400 294 225 195 170 12 469 326 239 183 150 132 14 393 273 201 154 128 112 16 336 233 PAGE4 Fascicle II.3 - Rec. E.421 171 131 112 98 18 292 202 149 114 95 80 20 256 178 131 100 85 70 30 149 104 76 60 50 42 40 96 67 50 38 30 24 50 Fascicle II.3 - Rec. E.421 PAGE1 64 44 33 25 20 16 Annex to Table 1/E.421 Examples of use of Table 1/E.421 Example 1 - It is estimated from previous results that a particular type of failure occurs on about 4% of calls. If it is required to confirm, with 95% confidence, that the existing failure rate is between 3% and 5% (i.e. ± 25% of 4%), then observations must be made on a random sample of 1536 calls. Example 2 - For an expected failure rate of 2%, observations must be made on a random sample of about 1200 calls (1225 in the table) to predict, with 95% confidence, that the true percentage is between 1.2% and 2.8% (i.e. ± 40% of 2%). This means that when 200 observations are taken over a period it is necessary to take the "rolling average" of conditions over six periods. The rate of failure for a number of categories important from the maintenance point of view is expected to be about 2%. Example 3 - After observations have been taken and the rate of failure in the sample has been calculated, the table may be used in a "backward" direction to give a rough indication of the accuracy of the result. Suppose that out of a sample of 1000 observations, there were 29 failures due to cause "X" and 15 failures due to cause "Y". The rates of failure in the sample due to X and Y, respectively, are then 2.9% and 1.5%. From the table, it is apparent from this sample of 1000 calls that the true rate of failure due to X has an accuracy of about ± 35% (i.e. is between 1.9% and 3.9%), and that due to Y has an accuracy of about ± 50% (i.e. is between 0.8% and 2.3%). 6 Exchange and analysis of the results of observations 6.1 Exchange of the results of observations The following periodicities are proposed for the exchange of results between Administrations: Table 1/E.422 or Table 2/E.422 - a monthly exchange is desirable; Table 1/E.423 or Table 2/E.423 - a quarterly exchange is desirable. Nevertheless, in the case of small groups of circuits (less than 20 circuits) the information should be exchanged after 200 observations have been made but never later than one year in any case; attention is drawn to the fact that less than 200 observations are of little value. Results of observations will be reported without delay: - to the Administrations and the network analysis point of the country where observations are carried out; - to the Administrations and the network analysis point of the other country (including transit Administrations and their network analysis point when involved). The benefits to be derived from service observations tend to decrease if there is any increase in the time taken to make the results available to those who can take action to bring about an improvement. The results of service observations according to Tables 1/E.422 and 1/E.423 should therefore be made available to the Administration in the countries of destination as soon as possible after completion of the observation period and in any case within six weeks. 6.2 Analysis of observation results An analysis of the results should be carried out in the country of origin as well as in the country of destination. Some Administrations have found it useful to distribute to other Administrations concerned, service observation statistics in the form of graphs. PAGE4 Fascicle II.3 - Rec. E.421