View Item 
        •   Utrecht University Student Theses Repository Home
        • UU Theses Repository
        • Theses
        • View Item
        •   Utrecht University Student Theses Repository Home
        • UU Theses Repository
        • Theses
        • View Item
        JavaScript is disabled for your browser. Some features of this site may not work without it.

        Browse

        All of UU Student Theses RepositoryBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

        Image classifiers in automated vehicles: An evaluation framework for robustness

        Thumbnail
        View/Open
        Thesis_lovis_heindrich.pdf (14.40Mb)
        Publication date
        2020
        Author
        Heindrich, L.
        Metadata
        Show full item record
        Summary
        This thesis reports the development and evaluation of a general safety evaluation procedure for image classifiers that can be applied to automated vehicles. Artificial Intelligence (AI) systems are widely used in automated vehicles, for example to process camera inputs. For the deployment of safety-critical systems, their safety assessment is of critical importance. However, current type approval and safety tests in cars are not suitable for evaluating machine learning components due to their black-box nature and general instability. To fill this research gap, I developed a methodological evaluation framework that tests the robustness of image classifiers. The testing procedure consists of two separate tests. The first test evaluates the ability to function in challenging and novel conditions. The second test evaluates the system’s behavior when confronted with out-of-distribution (OOD) examples. Evaluation measures for these tests are accuracy, separability of OOD examples, and overall quality of certainty estimates. I tested the framework on two different classifier architectures: a baseline Deep Convolutional Neural Network (DCNN) and Monte Carlo (MC) Dropout, which is used due to its more accurate certainty predictions. Results of the evaluation showed that different image transformations provide critical tests for both evaluating the classifier on challenging in-distribution images and realistic OOD examples. In this way, the evaluation framework can be used to identify weak points in the classifier’s robustness and show their unsuitability for safety-critical applications. Comparing the two classifiers with the evaluation shows the better generalization of MC Dropout, which was not detectable when evaluating with a traditional test set. The evaluation can ultimately be used to define acceptance criteria for the safe use of AI systems, for example, in type-approval for automated vehicles.
        URI
        https://studenttheses.uu.nl/handle/20.500.12932/37078
        Collections
        • Theses
        Utrecht university logo