Browse Source

Merge pull request #731 from jplag/refactor-e2e

Refactor End to End Tests
Timur Sağlam 4 years ago
parent
commit
758d4f5ece

+ 31 - 35
endtoend-testing/README.md

@@ -85,35 +85,34 @@ The internal structure consists of several `Option` records, each of which conta
 Thus the results can be kept apart from the other configurations. 
 The test results for the specified options are also specified in the object. This consists of the `ExpectedResult` record which contains the results of the detection.
 
-Here the herachie is as follows:
+Here the hierarchy is as follows:
 
 ```JSON
 [{
-      "options":{
-         "minimum_token_match":"int"
-      },
-      "tests":{
-         "languageIdentifier":{
-            "minimal_similarity":"float",
-            "maximum_similarity":"float",
-            "matched_token_number":"int"
-         },
-         "/..."
-      }
-   },
-   "options":{
-      "minimum_token_match":"int"
-   },
-   "tests":{
-      "languageIdentifier":{
-         "minimal_similarity":"float",
-         "maximum_similarity":"float",
-         "matched_token_number":"int"
-      },
-      {
-         "/..."
-      }
-   }
+  "options":{
+    "minimum_token_match":"int"
+  },
+  "tests":{
+    "languageIdentifier":{
+      "minimal_similarity":"float",
+      "maximum_similarity":"float",
+      "matched_token_number":"int"
+    },
+  "/..."
+  }
+},
+{ 
+  "options":{
+    "minimum_token_match":"int"
+  },
+  "tests":{
+    "languageIdentifier":{
+      "minimal_similarity":"float",
+      "maximum_similarity":"float",
+      "matched_token_number":"int"
+    },
+    "/..."
+  }
 }]
 ```
 
@@ -173,14 +172,11 @@ public void BubbleSortWithoutRecursion(Integer arr[]) {
 ```
 ### Copying Plagiarism To The Resources
 
-The plagiarisms created in [Creating The Plagiarism](#creating-the-plagiarism) must now be copied to the corresponding resources folder. It is important not to mix the languages of the plagiarisms or to copy the data into bottle resource paths.
+The plagiarisms created in [Creating The Plagiarism](#creating-the-plagiarism) need now to be copied to the corresponding resources folder. For each test suite, the resources must be placed in `JPlag/jplag.endToEndTesting/src/test/resources/languageTestFiles/<languageIdentifier>/<testSuiteIdentifier>`. For example, for the existing test suite `sortAlgo` of language `java`, the path is `JPlag/jplag.endToEndTesting/src/test/resources/languageTestFiles/java/sortAlgo`.
+It is important to note that the language identifier must match `Language#getIdentifier` to correctly load the language during testing.
 
-- At the path `JPlag/jplag.endToEndTesting/src/test/resources/languageTestFiles` a new folder for the language should be created if it does not already exist. For example `[...]/resources/languageTestFiles/JAVA`. If you have plagiarized several different code samples, you can also create additional subfolders under the newly created folder for example `[...]/resources/languageTestFiles/JAVA/sortAlgo`.
-
-It is important to note that the resource folder name must be the same as the language identifier name in JPlag/Language. 
-Otherwise, the language option cannot be parsed correctly to the enum-type which you can found in every language module under `Language.java` `IDENTIFIER`
-
-Once the tests have been run for the first time, the information for the tests is stored in the folder `../target/testing-directory-submission/LANGUAGE`.  This data can be copied to the path `[...]/resources/results/LANGUAGE`. Each subdirectory gets its result JSON file as `[...]/resources/results/LANGUAGE/TEST_SUITE_NAME.json`. Once the test data has been copied, the end-to-end tests can be successfully tested. As soon as a change in the detection takes place, the results will differ from the stored results and the tests will fail if the results have changed.
+To automatically generate expected results, the test in `EndToEndGeneratorTest` can be executed to generate a JSON result description file. This file has to be copied to `JPlag/jplag.endToEndTesting/src/test/resources/results/<languageIdentifier>/<testSuiteIdentifier>.json`.
+Once the test data has been copied, the end-to-end tests can be successfully tested. As soon as a change in the detection takes place, the results will differ from the stored results and the tests will fail if the results have changed.
 
 ### Extending The Comparison Value
 
@@ -202,8 +198,8 @@ public record ExpectedResult(
 
 ```JAVA
 //...
-            if (Float.compare(result.resultSimilarityMaximum(), jPlagComparison.maximalSimilarity()) != 0) {
-                addToValidationErrors("maximalSimilarity", String.valueOf(result.resultSimilarityMaximum()),
+            if (areDoublesDifferent(result.resultSimilarityMaximum(), jPlagComparison.maximalSimilarity())) {
+                addToValidationErrors("maximal similarity", String.valueOf(result.resultSimilarityMaximum()),
                         String.valueOf(jPlagComparison.maximalSimilarity()));
             }
 //...

+ 0 - 6
endtoend-testing/src/main/java/de/jplag/endtoend/constants/TestDirectoryConstants.java

@@ -11,12 +11,6 @@ public final class TestDirectoryConstants {
         // private constructor to prevent instantiation
     }
 
-    /**
-     * Create the complete path to the temporary result files. Here the temporary system path is extended with the
-     * "RESULT_DIRECTORY_NAME", which is predefined in this class.
-     */
-    public static final Path TEMPORARY_RESULT_DIRECTORY_NAME = Path.of("target", "testing-directory-temporary-result");
-
     /**
      * Base path to the saved results
      */

+ 7 - 95
endtoend-testing/src/main/java/de/jplag/endtoend/helper/FileHelper.java

@@ -2,16 +2,6 @@ package de.jplag.endtoend.helper;
 
 import java.io.File;
 import java.io.IOException;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.nio.file.StandardCopyOption;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.List;
-import java.util.Objects;
-import java.util.stream.Collectors;
-
-import de.jplag.endtoend.constants.TestDirectoryConstants;
 
 /**
  * Helper class to perform all necessary operations or functions on files or folders.
@@ -23,79 +13,14 @@ public class FileHelper {
     }
 
     /**
-     * Merges all contained filenames together without extension
-     * @param files whose names are to be merged
-     * @return merged filenames
-     */
-    public static String getEnclosedFileNamesFromCollection(Collection<File> files) {
-
-        return files.stream().map(File::getName).map(fileName -> fileName.substring(0, fileName.lastIndexOf('.'))).collect(Collectors.joining());
-    }
-
-    /**
-     * Load all possible languages in resource path
-     * @param directoryNames folder names for which the language options should be listed.
-     * @return list of all LanguageOptions included in the resource path
-     */
-    public static List<String> getLanguageOptionsFromPath(String[] directoryNames) {
-        return Arrays.stream(directoryNames).map(language -> language).filter(Objects::nonNull).toList();
-    }
-
-    /**
-     * @param directorieRoot path from which all folders should be loaded
-     * @return all folders found in the specified path
-     */
-    public static String[] getAllDirectoriesInPath(Path directorieRoot) {
-        return directorieRoot.toFile().list((dir, name) -> new File(dir, name).isDirectory());
-    }
-
-    /**
-     * Copies the passed filenames to a temporary path to use them in the tests
-     * @param classNames for which the test case is to be created
-     * @return paths created to the test submissions
-     * @throws IOException Exception can be thrown in cases that involve reading, copying or locating files.
+     * Returns the name of the passed file, trimming its file extension.
+     * @param file is the file to obtain the name from
+     * @return returns the name of the file without file extension
      */
-    public static String[] createNewTestCaseDirectory(String[] classNames) throws IOException {
-        // Copy the resources data to the temporary path
-        String[] returnSubmissionPath = new String[classNames.length];
-        for (int counter = 0; counter < classNames.length; counter++) {
-            Path originalPath = Path.of(classNames[counter]);
-            returnSubmissionPath[counter] = Path
-                    .of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString(), "submission" + (counter + 1)).toAbsolutePath()
-                    .toString();
-            Path copyPath = Path.of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString(), "submission" + (counter + 1),
-                    originalPath.getFileName().toString());
-
-            File directory = new File(copyPath.toString());
-            if (!directory.exists()) {
-                directory.mkdirs();
-            }
-            Files.copy(originalPath, copyPath, StandardCopyOption.REPLACE_EXISTING);
-        }
-        return returnSubmissionPath;
-    }
-
-    /**
-     * Delete directory with including files
-     * @param folder Path to a folder or file to be deleted. This happens recursively to the path
-     * @throws IOException if an I/O error occurs
-     */
-    public static void deleteCopiedFiles(File folder) throws IOException {
-        if (!folder.exists()) {
-            return;
-        }
-        File[] files = folder.listFiles();
-        if (files == null) { // some JVMs return null for empty dirs
-            return;
-        }
-        for (File file : files) {
-            if (file.isDirectory()) {
-                deleteCopiedFiles(file);
-            } else {
-                Files.delete(file.toPath());
-            }
-        }
-        Files.delete(folder.toPath());
+    public static String getFileNameWithoutFileExtension(File file) {
+        String name = file.getName();
+        int index = name.lastIndexOf('.');
+        return index == -1 ? name : name.substring(0, index);
     }
 
     /**
@@ -120,19 +45,6 @@ public class FileHelper {
         }
     }
 
-    /**
-     * @param resourcenPaths list of paths that lead to test resources
-     * @return all filenames contained in the paths
-     */
-    public static String[] loadAllTestFileNames(Path resourcenPaths) {
-        var files = resourcenPaths.toFile().listFiles();
-        String[] fileNames = new String[files.length];
-        for (int i = 0; i < files.length; i++) {
-            fileNames[i] = files[i].getName();
-        }
-        return fileNames;
-    }
-
     /**
      * @param file for which the exception text is to be created
      * @return exception text for the specified file

+ 0 - 71
endtoend-testing/src/main/java/de/jplag/endtoend/helper/JsonHelper.java

@@ -1,71 +0,0 @@
-package de.jplag.endtoend.helper;
-
-import java.io.IOException;
-import java.nio.file.Path;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.ObjectWriter;
-
-import de.jplag.endtoend.constants.TestDirectoryConstants;
-import de.jplag.endtoend.model.ResultDescription;
-
-/**
- * Helper class for serialization and deserialization of the used json format into the correct record classes. The
- * serialization/deserialization is enabled using the Jackson library. The record classes used can be found at
- * {@link de.jplag.endtoend.model}.
- */
-public class JsonHelper {
-    /**
-     * private constructor to prevent instantiation
-     */
-    private JsonHelper() {
-        // For Serialization
-    }
-
-    /**
-     * @param directoryName name to the result path
-     * @param languageIdentifier for which the results are to be loaded
-     * @return ResultDescription as serialized object
-     * @throws IOException is thrown for all problems that may occur while parsing the json file. This includes both reading
-     */
-    public static List<ResultDescription> getJsonModelListFromPath(String directoryName, String languageIdentifier) throws IOException {
-
-        Path jsonPath = Path.of(TestDirectoryConstants.BASE_PATH_TO_RESULT_JSON.toString(), languageIdentifier, directoryName + ".json");
-
-        if (jsonPath.toFile().exists() && jsonPath.toFile().length() > 0) {
-
-            return Arrays.asList(new ObjectMapper().readValue(jsonPath.toFile(), ResultDescription[].class));
-        } else {
-            return Collections.<ResultDescription>emptyList();
-        }
-    }
-
-    /**
-     * Saves the passed object as a json file to the given path
-     * @param resultDescriptionist list of elements to be saved
-     * @param directoryName path to the temporary storage location
-     * @param languageIdentifier for which the results should be stored
-     * @throws IOException Signals that an I/O exception of some sort has occurred. Thisclass is the general class of
-     * exceptions produced by failed orinterrupted I/O operations.
-     */
-    public static void writeJsonModelsToJsonFile(List<ResultDescription> resultDescriptionist, String directoryName, String languageIdentifier)
-            throws IOException {
-        // create an instance of DefaultPrettyPrinter
-        // new DefaultPrettyPrinter()
-        ObjectWriter writer = new ObjectMapper().writer().withDefaultPrettyPrinter();
-
-        Path temporaryDirectory = Path.of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString(), languageIdentifier,
-                directoryName + ".json");
-
-        FileHelper.createDirectoryIfItDoesNotExist(temporaryDirectory.getParent().toFile());
-        FileHelper.createFileIfItDoesNotExist(temporaryDirectory.toFile());
-
-        // convert book object to JSON file
-
-        writer.writeValue(temporaryDirectory.toFile(), resultDescriptionist.toArray());
-
-    }
-}

+ 17 - 49
endtoend-testing/src/main/java/de/jplag/endtoend/helper/TestSuiteHelper.java

@@ -1,16 +1,11 @@
 package de.jplag.endtoend.helper;
 
 import java.io.File;
-import java.io.IOException;
-import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashMap;
 import java.util.List;
-import java.util.Map;
 import java.util.stream.Collectors;
 
 import de.jplag.JPlagComparison;
+import de.jplag.Language;
 import de.jplag.Submission;
 import de.jplag.endtoend.constants.TestDirectoryConstants;
 
@@ -26,28 +21,6 @@ public class TestSuiteHelper {
         // For Serialization
     }
 
-    /**
-     * Loads all existing test data into a test structure. the complex structure consists of the LanguageOption and their
-     * data to be tested. These are divided into folders. for more information please read the README file
-     * @return mapped LanguageOption to the data under test
-     */
-    public static Map<String, Map<String, Path>> getAllLanguageResources() {
-        String[] languageDirectoryNames = FileHelper.getAllDirectoriesInPath(TestDirectoryConstants.BASE_PATH_TO_LANGUAGE_RESOURCES);
-        List<String> languageInPathList = FileHelper.getLanguageOptionsFromPath(languageDirectoryNames);
-
-        Map<String, Map<String, Path>> returnMap = new HashMap<>();
-
-        for (String languageIdentifier : languageInPathList) {
-            var tempMap = new HashMap<String, Path>();
-            var allDirectoriesInPath = FileHelper
-                    .getAllDirectoriesInPath(Path.of(TestDirectoryConstants.BASE_PATH_TO_LANGUAGE_RESOURCES.toString(), languageIdentifier));
-            Arrays.asList(allDirectoriesInPath).forEach(directory -> tempMap.put(Path.of(directory).toFile().getName(),
-                    Path.of(TestDirectoryConstants.BASE_PATH_TO_LANGUAGE_RESOURCES.toString(), languageIdentifier, directory)));
-            returnMap.put(languageIdentifier, tempMap);
-        }
-        return returnMap;
-    }
-
     /**
      * Creates a unique identifier from the submissions in the JPlagComparison object which is used to find the results in
      * the json files.
@@ -55,35 +28,30 @@ public class TestSuiteHelper {
      * @return unique identifier for test case recognition
      */
     public static String getTestIdentifier(JPlagComparison jPlagComparison) {
-
-        return List.of(jPlagComparison.firstSubmission(), jPlagComparison.secondSubmission()).stream().map(Submission::getFiles)
-                .map(FileHelper::getEnclosedFileNamesFromCollection).sorted().collect(Collectors.joining("-"));
+        return List.of(jPlagComparison.firstSubmission(), jPlagComparison.secondSubmission()).stream().map(Submission::getRoot)
+                .map(FileHelper::getFileNameWithoutFileExtension).sorted().collect(Collectors.joining("-"));
 
     }
 
     /**
-     * Creates the permutation of all data contained in the passed parameters and adds it to the given path.
-     * @param fileNames for which the permutations are needed
-     * @param path to which the permutations are to be copied
-     * @return all permutations of the specified files to the path specified
+     * Returns the file pointing to the directory of the submissions for the given language and result json. The result
+     * json's name is expected to be equal to the test suite identifier.
+     * @param language is the language for the tests
+     * @param resultJSON is the json containing the expected values
+     * @return returns the directory of the submissions
      */
-    public static List<String[]> getTestCases(String[] fileNames, Path path) {
-        ArrayList<String[]> testCases = new ArrayList<>();
-
-        for (int outerCounter = 0; outerCounter < fileNames.length; outerCounter++) {
-            for (int innerCounter = outerCounter + 1; innerCounter < fileNames.length; innerCounter++) {
-                testCases.add(new String[] {Path.of(path.toAbsolutePath().toString(), fileNames[outerCounter]).toString(),
-                        Path.of(path.toAbsolutePath().toString(), fileNames[innerCounter]).toString()});
-            }
-        }
-        return testCases;
+    public static File getSubmissionDirectory(Language language, File resultJSON) {
+        return getSubmissionDirectory(language, FileHelper.getFileNameWithoutFileExtension(resultJSON));
     }
 
     /**
-     * The copied data should be deleted after instance closure
-     * @throws IOException if an I/O error occurs
+     * Returns the file pointing to the directory of the submissions for the given language and test suite identifier as
+     * described in the Readme.md.
+     * @param language is the langauge for the tests
+     * @param testSuiteIdentifier is the test suite identifier of the tests
+     * @return returns the directory of the submissions
      */
-    public static void clear() throws IOException {
-        FileHelper.deleteCopiedFiles(new File(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString()));
+    public static File getSubmissionDirectory(Language language, String testSuiteIdentifier) {
+        return TestDirectoryConstants.BASE_PATH_TO_LANGUAGE_RESOURCES.resolve(language.getIdentifier()).resolve(testSuiteIdentifier).toFile();
     }
 }

+ 0 - 19
endtoend-testing/src/main/java/de/jplag/endtoend/model/ResultDescription.java

@@ -12,23 +12,4 @@ import com.fasterxml.jackson.annotation.JsonProperty;
  */
 public record ResultDescription(@JsonIgnore String languageIdentifier, @JsonProperty("options") Options options,
         @JsonProperty("tests") Map<String, ExpectedResult> identifierToResultMap) {
-
-    /**
-     * @param identifier for which the stored results are needed
-     * @return stored results as ExpectedResult object for the passed id
-     */
-    @JsonIgnore
-    public ExpectedResult getExpectedResultByIdentifier(String identifier) {
-        return identifierToResultMap.get(identifier);
-    }
-
-    /**
-     * Adds expected results to the existing list of the object. These results are stored in a map with the given
-     * identifier.
-     * @param identifier under which the results should be stored
-     * @param expectedResult expected results belonging to the identifier
-     */
-    public void putIdentifierToResultMap(String identifier, ExpectedResult expectedResult) {
-        identifierToResultMap.put(identifier, expectedResult);
-    }
 }

+ 87 - 0
endtoend-testing/src/test/java/de/jplag/endtoend/EndToEndGeneratorTest.java

@@ -0,0 +1,87 @@
+package de.jplag.endtoend;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
+
+import de.jplag.JPlag;
+import de.jplag.JPlagComparison;
+import de.jplag.JPlagResult;
+import de.jplag.Language;
+import de.jplag.LanguageLoader;
+import de.jplag.endtoend.constants.TestDirectoryConstants;
+import de.jplag.endtoend.helper.FileHelper;
+import de.jplag.endtoend.helper.TestSuiteHelper;
+import de.jplag.endtoend.model.ExpectedResult;
+import de.jplag.endtoend.model.Options;
+import de.jplag.endtoend.model.ResultDescription;
+import de.jplag.exceptions.ExitException;
+import de.jplag.options.JPlagOptions;
+
+/**
+ * Test class for automatically generating the json file describing the expected results. To generate a result json,
+ * adapt the three constants to your requirements and enable the test case.
+ */
+class EndToEndGeneratorTest {
+    private static final String LANGUAGE_IDENTIFIER = "java";
+    private static final String TEST_SUITE_IDENTIFIER = "sortAlgo";
+    private static final List<Options> OPTIONS = List.of(new Options(3), new Options(9));
+
+    private static final Logger logger = LoggerFactory.getLogger(EndToEndGeneratorTest.class);
+
+    @Disabled("only enable to generate result json file")
+    @Test
+    void generateResultJson() throws ExitException, IOException {
+        Language language = LanguageLoader.getLanguage(LANGUAGE_IDENTIFIER).orElseThrow();
+        File submissionDirectory = TestSuiteHelper.getSubmissionDirectory(language, TEST_SUITE_IDENTIFIER);
+        List<ResultDescription> resultDescriptions = new ArrayList<>();
+        for (var option : OPTIONS) {
+            JPlagOptions jplagOptions = new JPlagOptions(language, Set.of(submissionDirectory), Set.of())
+                    .withMinimumTokenMatch(option.minimumTokenMatch());
+            JPlagResult jplagResult = new JPlag(jplagOptions).run();
+            List<JPlagComparison> jPlagComparisons = jplagResult.getAllComparisons();
+            Map<String, ExpectedResult> expectedResults = jPlagComparisons.stream()
+                    .collect(Collectors.toMap(TestSuiteHelper::getTestIdentifier, comparison -> new ExpectedResult(comparison.minimalSimilarity(),
+                            comparison.maximalSimilarity(), comparison.getNumberOfMatchedTokens())));
+            resultDescriptions.add(new ResultDescription(language.getIdentifier(), option, expectedResults));
+        }
+        File outputFile = writeJsonModelsToJsonFile(resultDescriptions, TEST_SUITE_IDENTIFIER, LANGUAGE_IDENTIFIER);
+        logger.info("result JSON written to file '{}'", outputFile);
+    }
+
+    /**
+     * Saves the passed object as a json file to the file identified by the test suite and language. Returns that file.
+     * @param resultDescriptions list of elements to be saved
+     * @param testSuiteIdentifier identifier of the test suite
+     * @param languageIdentifier identifier of the language
+     * @throws IOException Signals that an I/O exception of some sort has occurred. Thisclass is the general class of
+     * exceptions produced by failed orinterrupted I/O operations.
+     */
+    private static File writeJsonModelsToJsonFile(List<ResultDescription> resultDescriptions, String testSuiteIdentifier, String languageIdentifier)
+            throws IOException {
+        ObjectWriter writer = new ObjectMapper().writer().withDefaultPrettyPrinter();
+        File outputFile = TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.resolve(languageIdentifier)
+                .resolve(testSuiteIdentifier + ".json").toFile();
+
+        FileHelper.createDirectoryIfItDoesNotExist(outputFile.getParentFile());
+        FileHelper.createFileIfItDoesNotExist(outputFile);
+
+        // convert book object to JSON file
+
+        writer.writeValue(outputFile, resultDescriptions.toArray());
+        return outputFile;
+
+    }
+}

+ 76 - 197
endtoend-testing/src/test/java/de/jplag/endtoend/EndToEndSuiteTest.java

@@ -1,36 +1,34 @@
 package de.jplag.endtoend;
 
+import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
 import java.io.File;
 import java.io.IOException;
-import java.nio.file.Path;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.Collection;
-import java.util.HashMap;
+import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Optional;
 import java.util.Set;
-import java.util.StringJoiner;
 import java.util.stream.Collectors;
 
-import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.DynamicContainer;
 import org.junit.jupiter.api.DynamicTest;
 import org.junit.jupiter.api.TestFactory;
 
+import com.fasterxml.jackson.databind.ObjectMapper;
+
 import de.jplag.JPlag;
 import de.jplag.JPlagComparison;
 import de.jplag.JPlagResult;
+import de.jplag.Language;
 import de.jplag.LanguageLoader;
+import de.jplag.endtoend.constants.TestDirectoryConstants;
 import de.jplag.endtoend.helper.FileHelper;
-import de.jplag.endtoend.helper.JsonHelper;
 import de.jplag.endtoend.helper.TestSuiteHelper;
 import de.jplag.endtoend.model.ExpectedResult;
-import de.jplag.endtoend.model.Options;
 import de.jplag.endtoend.model.ResultDescription;
 import de.jplag.exceptions.ExitException;
 import de.jplag.options.JPlagOptions;
@@ -40,146 +38,88 @@ import de.jplag.options.JPlagOptions;
  * plagiarism in the Java language and to be able to roughly categorize them. The plagiarism is compared with the
  * original class. The results are compared with the results from previous tests and changes are detected.
  */
-public class EndToEndSuiteTest {
+class EndToEndSuiteTest {
     private static final double EPSILON = 1E-8;
-    // Language -> directory names and Paths
-    private Map<String, Map<String, Path>> languageToTestCaseMapper;
-
-    private List<Options> options;
-
-    private static Map<String, List<ResultDescription>> temporaryResultList;
-    private static List<String> validationErrors;
-    private static String languageIdentifier;
-
-    public EndToEndSuiteTest() throws IOException {
-        // Loading the test resources
-        languageToTestCaseMapper = TestSuiteHelper.getAllLanguageResources();
-        // creating the temporary lists for the test run
-        validationErrors = new ArrayList<>();
-        temporaryResultList = new HashMap<>();
-        // creating options object for the testSuite
-        setRunOptions();
-    }
-
-    /**
-     * creating the required options object for the endToEnd tests
-     */
-    private void setRunOptions() {
-        options = new ArrayList<>();
-        options.add(new Options(9));
-        options.add(new Options(3));
-    }
-
-    /**
-     * Creates the result json files based on the current test results after the test run.
-     * @throws IOException is thrown for all problems that may occur while parsing the json file. This includes both reading
-     */
-    @AfterAll
-    public static void tearDown() throws IOException {
-        for (var resultDescriptionItem : temporaryResultList.entrySet()) {
-            JsonHelper.writeJsonModelsToJsonFile(resultDescriptionItem.getValue(), resultDescriptionItem.getKey(), languageIdentifier);
-        }
-    }
 
     /**
      * Creates the test cases over all language options for which data is available and the current test options.
      * @return dynamic test cases across all test data and languages
-     * @throws IOException is thrown for all problems that may occur while parsing the json file. This includes both reading
+     * @throws IOException is thrown for all problems that may occur while parsing the json file.
      */
     @TestFactory
-    Collection<DynamicTest> dynamicOverAllTest() throws IOException {
-        for (Entry<String, Map<String, Path>> languageMap : languageToTestCaseMapper.entrySet()) {
-            String currentLanguageIdentifier = languageMap.getKey();
-            for (Entry<String, Path> languagePaths : languageMap.getValue().entrySet()) {
-                String[] fileNames = FileHelper.loadAllTestFileNames(languagePaths.getValue());
-                var testCases = TestSuiteHelper.getTestCases(fileNames, languagePaths.getValue());
-                var testCollection = new ArrayList<DynamicTest>();
-                String directoryName = languagePaths.getValue().getFileName().toString();
-                List<ResultDescription> tempResult = JsonHelper.getJsonModelListFromPath(directoryName, currentLanguageIdentifier);
-                languageIdentifier = currentLanguageIdentifier;
-                for (Options option : options) {
-                    for (var testCase : testCases) {
-                        Optional<ResultDescription> currentResultDescription = tempResult.stream().filter(x -> x.options().equals(option))
-                                .findFirst();
-                        testCollection.add(DynamicTest.dynamicTest(getTestCaseDisplayName(option, languageIdentifier, testCase), () -> {
-                            runTests(directoryName, option, currentLanguageIdentifier, testCase, currentResultDescription);
-                        }));
-                    }
+    Collection<DynamicContainer> dynamicOverAllTest() throws IOException, ExitException {
+        File resultsDirectory = TestDirectoryConstants.BASE_PATH_TO_RESULT_JSON.toFile();
+        File[] languageDirectories = resultsDirectory.listFiles(File::isDirectory);
+        List<DynamicContainer> allTests = new LinkedList<>();
+        for (File languageDirectory : languageDirectories) {
+            Language language = LanguageLoader.getLanguage(languageDirectory.getName()).orElseThrow();
+            File[] resultJsons = languageDirectory.listFiles(file -> !file.isDirectory() && file.getName().endsWith(".json"));
+            List<DynamicContainer> languageTests = new LinkedList<>();
+            for (File resultJson : resultJsons) {
+                List<DynamicContainer> testContainers = new LinkedList<>();
+                ResultDescription[] results = new ObjectMapper().readValue(resultJson, ResultDescription[].class);
+                for (var result : results) {
+                    var testCases = generateTestsForResultDescription(resultJson, result, language);
+                    testContainers.add(DynamicContainer.dynamicContainer("MTM: " + result.options().minimumTokenMatch(), testCases));
                 }
-                return testCollection;
+                languageTests.add(DynamicContainer.dynamicContainer(FileHelper.getFileNameWithoutFileExtension(resultJson), testContainers));
             }
+            allTests.add(DynamicContainer.dynamicContainer(language.getIdentifier(), languageTests));
         }
-        return null;
+        return allTests;
     }
 
     /**
-     * Superordinate test function to be able to continue to check all data to be tested in case of failed tests
-     * @param directoryName name of the current tested directory
-     * @param option for the current test run
-     * @param currentLanguageIdentifier current JPlag language option
-     * @param testFiles files to be tested
-     * @param currentResultDescription results stored for the test data
-     * @throws IOException Signals that an I/O exception of some sort has occurred. Thisclass is the general class of
-     * exceptions produced by failed orinterrupted I/O operations
-     * @throws ExitException Exceptions for problems during the execution of JPlag that lead to an preemptive exit.
+     * Generates test cases for each test described in the provided result object.
+     * @param resultJson is the file of the result json
+     * @param result is one test suite configuration of the deserialized {@code resultJson}
+     * @param language is the language to run JPlag with
+     * @return a collection of test cases, each validating one {@link JPlagResult} against its {@link ExpectedResult}
+     * counterpart
      */
-    private void runTests(String directoryName, Options option, String currentLanguageIdentifier, String[] testFiles,
-            Optional<ResultDescription> currentResultDescription) throws IOException, ExitException {
-        try {
-            ResultDescription currentResult = currentResultDescription.orElse(null);
-            runJPlagTestSuite(directoryName, option, currentLanguageIdentifier, testFiles, currentResult);
-        } finally {
-            validationErrors.clear();
-            TestSuiteHelper.clear();
-        }
+    private Collection<DynamicTest> generateTestsForResultDescription(File resultJson, ResultDescription result, Language language)
+            throws ExitException {
+        File submissionDirectory = TestSuiteHelper.getSubmissionDirectory(language, resultJson);
+        JPlagOptions jplagOptions = new JPlagOptions(language, Set.of(submissionDirectory), Set.of())
+                .withMinimumTokenMatch(result.options().minimumTokenMatch());
+        JPlagResult jplagResult = new JPlag(jplagOptions).run();
+        Map<String, JPlagComparison> jPlagComparisons = jplagResult.getAllComparisons().stream()
+                .collect(Collectors.toMap(it -> TestSuiteHelper.getTestIdentifier(it), it -> it));
+        assertEquals(result.identifierToResultMap().size(), jPlagComparisons.size(), "different number of results and expected results");
+
+        return result.identifierToResultMap().keySet().stream().map(identifier -> {
+            JPlagComparison comparison = jPlagComparisons.get(identifier);
+            ExpectedResult expectedResult = result.identifierToResultMap().get(identifier);
+            return generateTest(identifier, expectedResult, comparison);
+        }).toList();
     }
 
     /**
-     * EndToEnd test for the passed objects
-     * @param directoryName name of the current tested directory
-     * @param options for the current test run
-     * @param languageIdentifier current JPlag language option
-     * @param testFiles files to be tested
-     * @param currentResultDescription results stored for the test data
-     * @throws IOException Signals that an I/O exception of some sort has occurred. Thisclass is the general class of
-     * exceptions produced by failed orinterrupted I/O operations
-     * @throws ExitException Exceptions for problems during the execution of JPlag that lead to an preemptive exit.
+     * Generates a test case validating the passed result by comparing it to the expected result values.
+     * @param name is the name of the test case.
+     * @param expectedResult contains all expected result values.
+     * @param result is the comparison object generated from running JPlag.
      */
-    private void runJPlagTestSuite(String directoryName, Options options, String languageIdentifier, String[] testFiles,
-            ResultDescription currentResultDescription) throws IOException, ExitException {
-        String[] submissionPath = FileHelper.createNewTestCaseDirectory(testFiles);
-
-        var language = LanguageLoader.getLanguage(languageIdentifier).orElseThrow();
-        JPlagOptions jplagOptions = new JPlagOptions(language, Arrays.stream(submissionPath).map(path -> new File(path)).collect(Collectors.toSet()),
-                Set.of()).withMinimumTokenMatch(options.minimumTokenMatch());
-        JPlagResult jplagResult = new JPlag(jplagOptions).run();
-
-        List<JPlagComparison> currentJPlagComparison = jplagResult.getAllComparisons();
-
-        for (JPlagComparison jPlagComparison : currentJPlagComparison) {
-            String identifier = TestSuiteHelper.getTestIdentifier(jPlagComparison);
-            addToTemporaryResultMap(directoryName, options, jPlagComparison, languageIdentifier);
-
-            assertNotNull(currentResultDescription, "No stored result could be found for the current LanguageOption! " + options);
-
-            ExpectedResult result = currentResultDescription.getExpectedResultByIdentifier(TestSuiteHelper.getTestIdentifier(jPlagComparison));
-            assertNotNull(result, "No stored result could be found for the identifier! " + identifier);
-
-            if (areDoublesDifferent(result.resultSimilarityMinimum(), jPlagComparison.minimalSimilarity())) {
-                addToValidationErrors("minimalSimilarity", String.valueOf(result.resultSimilarityMinimum()),
-                        String.valueOf(jPlagComparison.minimalSimilarity()));
+    private DynamicTest generateTest(String name, ExpectedResult expectedResult, JPlagComparison result) {
+        return DynamicTest.dynamicTest(name, () -> {
+            assertNotNull(result, "No comparison result could be found");
+
+            List<String> validationErrors = new ArrayList<>();
+            if (areDoublesDifferent(expectedResult.resultSimilarityMinimum(), result.minimalSimilarity())) {
+                validationErrors.add(formattedValidationError("minimal similarity", String.valueOf(expectedResult.resultSimilarityMinimum()),
+                        String.valueOf(result.minimalSimilarity())));
             }
-            if (areDoublesDifferent(result.resultSimilarityMaximum(), jPlagComparison.maximalSimilarity())) {
-                addToValidationErrors("maximalSimilarity", String.valueOf(result.resultSimilarityMaximum()),
-                        String.valueOf(jPlagComparison.maximalSimilarity()));
+            if (areDoublesDifferent(expectedResult.resultSimilarityMaximum(), result.maximalSimilarity())) {
+                validationErrors.add(formattedValidationError("maximal similarity", String.valueOf(expectedResult.resultSimilarityMaximum()),
+                        String.valueOf(result.maximalSimilarity())));
             }
-            if (result.resultMatchedTokenNumber() != jPlagComparison.getNumberOfMatchedTokens()) {
-                addToValidationErrors("numberOfMatchedTokens", String.valueOf(result.resultMatchedTokenNumber()),
-                        String.valueOf(jPlagComparison.getNumberOfMatchedTokens()));
+            if (expectedResult.resultMatchedTokenNumber() != result.getNumberOfMatchedTokens()) {
+                validationErrors.add(formattedValidationError("number of matched tokens", String.valueOf(expectedResult.resultMatchedTokenNumber()),
+                        String.valueOf(result.getNumberOfMatchedTokens())));
             }
 
-            assertTrue(validationErrors.isEmpty(), createValidationErrorOutout());
-        }
+            assertTrue(validationErrors.isEmpty(), createValidationErrorOutput(validationErrors));
+        });
     }
 
     private boolean areDoublesDifferent(double d1, double d2) {
@@ -187,82 +127,21 @@ public class EndToEndSuiteTest {
     }
 
     /**
-     * Creates the display message for failed tests
+     * Creates the display message for a result value validation error.
      * @param valueName Name of the failed test object
-     * @param currentValue current test values
-     * @param expectedValue expected test values
+     * @param actualValue actual test value
+     * @param expectedValue expected test value
      */
-    private void addToValidationErrors(String valueName, String currentValue, String expectedValue) {
-        validationErrors.add(valueName + " was " + currentValue + " but expected " + expectedValue);
+    private String formattedValidationError(String valueName, String actualValue, String expectedValue) {
+        return valueName + " was " + actualValue + " but expected " + expectedValue;
     }
 
     /**
-     * Creates the display info from the current failed test results
+     * Creates the display info from the passed failed test results
      * @return formatted text for the failed comparative values of the current test
      */
-    private String createValidationErrorOutout() {
-        StringJoiner joiner = new StringJoiner(System.lineSeparator());
-        joiner.add(""); // empty line at start
-        joiner.add("There were <" + validationErrors.size() + "> validation error(s):");
-
-        validationErrors.stream().forEach(errorLine -> joiner.add(errorLine));
-        joiner.add(""); // line break at the end
-
-        return joiner.toString();
-    }
-
-    /**
-     * Add or create the current test results in a temporary list to be able to save them later.
-     * @param directoryName name of the current tested directory
-     * @param options for the current test run
-     * @param jPlagComparison current test results
-     * @param languageIdentifier current JPlag language option
-     */
-    private void addToTemporaryResultMap(String directoryName, Options options, JPlagComparison jPlagComparison, String languageIdentifier) {
-        var element = temporaryResultList.get(directoryName);
-
-        if (element != null) {
-            for (var item : element) {
-                if (item.options().equals(options)) {
-                    item.putIdentifierToResultMap(TestSuiteHelper.getTestIdentifier(jPlagComparison), new ExpectedResult(
-                            jPlagComparison.minimalSimilarity(), jPlagComparison.maximalSimilarity(), jPlagComparison.getNumberOfMatchedTokens()));
-                    return;
-                }
-            }
-            Map<String, ExpectedResult> temporaryHashMap = new HashMap<>();
-            temporaryHashMap.put(TestSuiteHelper.getTestIdentifier(jPlagComparison), new ExpectedResult(jPlagComparison.minimalSimilarity(),
-                    jPlagComparison.maximalSimilarity(), jPlagComparison.getNumberOfMatchedTokens()));
-            element.add(new ResultDescription(languageIdentifier, options, temporaryHashMap));
-        } else {
-            var temporaryNewResultList = new ArrayList<ResultDescription>();
-            Map<String, ExpectedResult> temporaryHashMap = new HashMap<>();
-            temporaryHashMap.put(TestSuiteHelper.getTestIdentifier(jPlagComparison), new ExpectedResult(jPlagComparison.minimalSimilarity(),
-                    jPlagComparison.maximalSimilarity(), jPlagComparison.getNumberOfMatchedTokens()));
-
-            temporaryNewResultList.add(new ResultDescription(languageIdentifier, options, temporaryHashMap));
-
-            temporaryResultList.put(directoryName, temporaryNewResultList);
-        }
-    }
-
-    /**
-     * Creates the name of the test for better assignment and readability Pattern: Language: (option values)
-     * filename1-filename2
-     * @param option under which the current test run
-     * @param languageIdentifier current language used in the test
-     * @param testFiles test data for assigning by filename
-     * @return display name for the individual tests
-     */
-    private String getTestCaseDisplayName(Options option, String languageIdentifier, String[] testFiles) {
-        StringBuilder stringBuilder = new StringBuilder();
-        stringBuilder.append("(" + String.valueOf(option.minimumTokenMatch()) + ")");
-        for (int counter = 0; counter < testFiles.length; counter++) {
-            String fileName = Path.of(testFiles[counter]).toFile().getName();
-            stringBuilder.append(fileName.substring(0, fileName.lastIndexOf('.')));
-            if (counter + 1 < testFiles.length) {
-                stringBuilder.append("-");
-            }
-        }
-        return languageIdentifier + ": " + stringBuilder.toString();
+    private String createValidationErrorOutput(List<String> validationErrors) {
+        return "There were " + validationErrors.size() + " validation error(s):" + System.lineSeparator()
+                + String.join(System.lineSeparator(), validationErrors);
     }
 }