文档

后端接口/Java-Service.md

@@ -169,3 +169,319 @@
 
 高级方法，使用了scala编写的FeatureProject类，暂时先忽略。不过，这个方法放在configService里面真的好吗？
 
+# 4. ModelService
+
+```java
+    public int addModel(ModelVO modelVO){
+        Model modelInfo = new Model();
+        modelInfo.setFileInfoId(Integer.parseInt(modelVO.getFileInfoId()));
+        modelInfo.setConfigId(Integer.parseInt(modelVO.getConfigId()));
+        modelInfo.setModelName(modelVO.getModelName());
+        modelInfo.setModelTypeId(Integer.parseInt(modelVO.getModelTypeId()));
+        modelInfo.setTrained(false);
+        modelInfo.setResOfModel(null);
+        modelInfo.setModelPath(null);
+        modelInfo.setArguments(JSON.toJSONString(modelVO.getArguments()));
+        return modelDao.save(modelInfo).getId();
+    }
+```
+
+将modelVO的属性注入到modelInfo中，设置trained属性为false，resofModel和modelPath为null，并保存。
+
+```java
+    public int addDlModel(DLModelVO dlModelVO) {
+        DLModel dlModel = new DLModel();
+        dlModel.setModelTypeId(Integer.parseInt(dlModelVO.getModelTypeId()));
+        dlModel.setPicFileId(Integer.parseInt(dlModelVO.getPicFileId()));
+        dlModel.setModelName(dlModelVO.getModelName());
+        dlModel.setTrained(false);
+        dlModel.setResOfModel(null);
+        dlModel.setModelPath(null);
+        dlModel.setArguments(JSON.toJSONString(dlModelVO.getArguments()));
+        return dlModelDao.save(dlModel).getId();
+    }
+```
+
+将dlModelVO的属性注入到dlModel中，设置trained属性为false，resofModel和modelPath为null，并保存。
+
+```java
+public List<ModelPojo> getAllModelByUserId(int userId){
+        List<ModelPojo> rsl = configDao.findByUserId(userId).stream().map((config) -> {
+            int configId = config.getId();
+            String configName = config.getConfName();
+            List<ModelPojo> modelPerConfig = modelDao.findByConfigId(configId).stream().map((model) -> {
+                ModelType modelType =  modelTypeDao.findById(model.getModelTypeId());
+                int flag = -1;
+                if(model.isTrained() && model.getResOfModel()!=null){
+                    flag = 1;
+                }else if(model.isTrained() && model.getResOfModel()==null){
+                    flag = 0;
+                }
+                return new ModelPojo(String.valueOf(configId),
+                        configName,
+                        String.valueOf(model.getModelTypeId()),
+                        modelType != null ? modelType.getModelTypeName() : "",
+                        modelType != null ? modelType.getModelDetailName() : "",
+                        modelType != null ? modelType.getModelDetailDes() : "",
+                        modelType != null ? modelType.getModelDes() : "",
+                        String.valueOf(model.getId()),
+                        model.getModelName(),
+                        flag,
+                        new Json2Object().Json2HashMap(model.getArguments()),
+                        new Json2Object().Json2HashMap(model.getResOfModel()));
+                }).collect(Collectors.toList());
+            return modelPerConfig;
+        }).reduce(new ArrayList<>(), (listA, listB) -> {
+            listA.addAll(listB);
+            return listA;
+        });
+        return rsl;
+    }
+```
+
+对于userId，找到该用户使用的所有config，对于每个找到的config，找到使用该config的model，并将该model属性进行填充，最后将所有的modelPojo合并返回。
+
+```java
+    public List<DLGeneralModelPojo> getAllGeneralModelByUserId(int userId){
+        List<PicFile> picFiles = pictureFileDao.findByUserId(userId);
+        List<DLGeneralModelPojo> result = picFiles.stream().map(picFile -> {
+            List<DLModel> dlModels = dlModelDao.findByPicFileId(picFile.getId());
+            return dlModels.stream()
+                    .map(dlModel -> new DLGeneralModelPojo(String.valueOf(dlModel.getId()), dlModel.getModelName(), String.valueOf(picFile.getId()), picFile.getFileName(),dlModel.isTrained()
+                            ,dlModel.getResOfModel()!=null))
+                    .collect(Collectors.toList());
+        }).reduce(new ArrayList<>(), (listA, listB) -> {
+            listA.addAll(listB);
+            return listA;
+        });
+        return result;
+    }
+```
+
+对于userId，找到该用户所有的picFiles，对每个picFile，找到使用该picFile的DLModel，封装为DLGeneralModelPojo，合并后返回。
+
+```java
+    public DLModelPojo getDLModelDetailById(int modelId) {
+        DLModel dlModel = dlModelDao.findById(modelId);
+        return new DLModelPojo(String.valueOf(dlModel.getModelTypeId()),String.valueOf(dlModel.getPicFileId()),
+                dlModel.getModelName(),dlModel.isTrained(),dlModel.getModelPath(),new Json2Object().Json2HashMap(dlModel.getResOfModel())
+        ,new Json2Object().Json2HashMap(dlModel.getArguments()));
+    }
+```
+
+对应某个modelId找到其具体信息并封装返回。
+
+```java
+    public List<ModelType> getModelTypeByModelTypeName(String modelTypeName) {
+        return modelTypeDao.findByModelTypeName(modelTypeName);
+    }
+```
+
+通过modelTypeName找到具体的ModelType信息并返回。
+
+```java
+    public boolean changeModel(int modelId, String modelName, int modelTypeId){
+        Model model = modelDao.findById(modelId);
+        model.setModelName(modelName);
+        model.setModelTypeId(modelTypeId);
+        modelDao.save(model);
+        return true;
+    }
+```
+
+修改某个Model实体并保存。
+
+```java
+    public boolean changeStatusOfTrained(int modelId, boolean trained){
+        Model model =  modelDao.findById(modelId);
+        model.setTrained(trained);
+        modelDao.save(model);
+        return true;
+    }
+```
+
+修改某个model训练状态。
+
+```java
+    public boolean updateResOfModel(int modelId, HashMap<String, String> resOfModel, Object model){
+        Model modelOfGet =  modelDao.findById(modelId);
+        //log.info("\n\nmodel:"+JSON.toJSONString(model)+"\n\n");
+        //T.B.D.
+        //modelOfGet.setModel(model);
+        modelOfGet.setResOfModel(JSON.toJSONString(resOfModel));
+        modelOfGet.setTrained(true);
+        log.info("\n\nstore model result: "+modelOfGet.getResOfModel());
+        modelDao.save(modelOfGet);
+        return true;
+    }
+```
+
+修改某个模型的resofModel，说起来，这个model你们到底更不更新？T.B.D.还没决定？
+
+```java
+public boolean trainTheModel(int modelId) throws Exception{
+
+        //change training flag to true
+        changeStatusOfTrained(modelId, true);
+
+        Model trainingModel = modelDao.findById(modelId);
+        ModelType modelType = modelTypeDao.findById(trainingModel.getModelTypeId());
+        Config configInfo = configDao.findById(trainingModel.getConfigId());
+        List<String> features = JSONObject.parseArray(configInfo.getFieldIds(),String.class).stream().map(fieldId ->
+                headerInfoDao.findById(Integer.parseInt(fieldId)).getFieldName()).collect(Collectors.toList());
+
+        //1. transfer file to libsvm
+        log.info("\n\nTransfer to LibSVM...\n\n");
+        String keyFeature = features.get(features.size()-1);
+        features.remove(features.size()-1);
+        int featureNum = features.size();
+        String libsvm = libsvmAdapter.parquet2Libsvm(configInfo.getFileInfoId(), keyFeature, features.toArray(new String[0]));
+
+        String modelDetailName = modelType.getModelDetailName();
+        HashMap<String,String> modelArguments = new Json2Object().Json2HashMap(trainingModel.getArguments());
+        boolean rsl = false;
+        try {
+            switch (modelDetailName) {
+                case "LogisticRegression":
+            /*
+            * maxIter: Int,
+                 regParam: Float,
+                 elasticNetParam: Float
+            * */
+                    LogisticRegressionModel lrModel = lrClassification.lrTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("MaxIter")),
+                            Float.parseFloat(modelArguments.get("RegParam")),
+                            Float.parseFloat(modelArguments.get("ElasticNetParam")));
+                    log.info("\n\nstore Model result\n\n");
+                    HashMap<String,String> lrRst = lrClassification.lrResult(lrModel);
+
+                    log.info("\n\nupdate result of model\n\n");
+                    rsl = updateResOfModel(modelId, lrRst, lrModel);
+                    break;
+                case "DecisionTree":
+            /*
+            * maxIter: Int,
+                 regParam: Float,
+                 elasticNetParam: Float
+            * */
+                    PipelineModel dtModel = dtClassification.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingSetOccupy")));
+                    HashMap<String,String> dtResult = dtClassification.dtResult(dtModel, modelId);
+
+                    rsl = updateResOfModel(modelId, dtResult, dtModel);
+                    break;
+                case "RandomForest":
+                    /*
+                * maxIter: Int,
+                 regParam: Float,
+                 elasticNetParam: Float
+            * */
+
+                    PipelineModel rfModel = rfClassification.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingDataSetOccupy")));
+                    HashMap<String,String> rfResult = rfClassification.dtResult(rfModel, modelId);
+
+                    rsl = updateResOfModel(modelId,rfResult,rfModel);
+                    break;
+                case "GBDT":
+                    /*
+                * maxIter: Int,
+                 regParam: Float,
+                 elasticNetParam: Float
+            * */
+
+                    PipelineModel gbtModel = gdbtClassification.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingDataSetOccupy")));
+                    HashMap<String,String> gbtResult = gdbtClassification.dtResult(gbtModel, modelId);
+
+                    rsl = updateResOfModel(modelId,gbtResult,gbtModel);
+                    break;
+
+                case "K-Means":
+                    KMeansModel kmeansModel = kmeansCluster.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("NumOfCluster")),
+                            Integer.parseInt(modelArguments.get("Seed")));
+                    HashMap<String,String> kmeansResult = kmeansCluster.dtResult(kmeansModel);
+                    rsl = updateResOfModel(modelId,kmeansResult,kmeansModel);
+                    break;
+
+                case "MultilayerPerceptronClassifier":
+                    PipelineModel mlpc = mulPerClassification.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingDataSetOccupy")), featureNum);
+                    HashMap<String,String> mlpcResult = mulPerClassification.dtResult(mlpc, modelId);
+
+                    rsl = updateResOfModel(modelId,mlpcResult,mlpc);
+                    break;
+
+                case "NaiveBayes":
+                    PipelineModel nby = naiveBayesClassification.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingDataSetOccupy")));
+                    HashMap<String,String> nbyResult = naiveBayesClassification.dtResult(nby, modelId);
+
+                    rsl = updateResOfModel(modelId,nbyResult,nby);
+                    break;
+
+                case "RandomForestRegression":
+                    PipelineModel rfReg = rfRegression.dtTraining(libsvm,
+                            Integer.parseInt(modelArguments.get("Category")),
+                            Float.parseFloat(modelArguments.get("TrainingDataSetOccupy"))
+                            );
+                    HashMap<String,String> rfRegResult = rfRegression.dtResult(rfReg);
+
+                    rsl = updateResOfModel(modelId,rfRegResult,rfReg);
+                    break;
+                default:
+                    rsl = false;
+            }
+        } catch (Exception e) {
+            e.printStackTrace();
+            changeStatusOfTrained(modelId, false);
+        }
+        if(!rsl){
+            changeStatusOfTrained(modelId,false);
+        }
+        return rsl;
+    }
+```
+
+高级方法，首先修改该模型状态为已训练。然后获取该模型的类型名称、配置、特征。
+
+然后将配置对应的文件从parquet文件转换为libsvm文件。
+
+> Twitter在Hadoop上面设计一种新的列式存储格式，这种格式可以保存包含嵌套结构的数据，即Parquet文件：列式存储一个类型包含嵌套结构的数据集。
+>
+> Spark通常使用libsvm文件作为输入数据，通常用于保存稀疏数据。
+
+然后通过modelDetailName来分别训练模型。
+
+如果没有训练成功，则将模型状态返回为未训练。
+
+而且，纵观整个modelService，是不是都没有涉及到model的保存？好歹保存一下modelPath啊喂！
+
+```java
+    public boolean deleteModel(int modelId) {
+        Model model =  modelDao.findById(modelId);
+        if(model == null){
+            return false;
+        }
+        modelDao.delete(modelDao.findById(modelId));
+        return true;
+    }
+```
+
+删除某个模型。
+
+```java
+    public List<Map<String, String>> useModel(int fileId, int modelId) throws Exception{
+        List<Map<String,String>> res = fitTestData.transformData(fileId,modelId);
+        fileService.deleteFile(fileId);
+        return res;
+    }
+```
+
+高级方法，对某个模型使用某个文件来获得结果，最后删除这个文件。（为什么？）