remove redundant

VideoPose3D/CODE_OF_CONDUCT.md

@@ -1,2 +0,0 @@
-# Code of Conduct
-Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please [read the full text](https://code.facebook.com/codeofconduct) so that you can understand what actions will and will not be tolerated.

VideoPose3D/CONTRIBUTING.md

@@ -1,30 +0,0 @@
-# Contributing
-We want to make contributing to this project as easy and transparent as
-possible.
-
-## Pull Requests
-We actively welcome your pull requests.
-
-1. Fork the repo and create your branch from `master`.
-2. If you've added code that should be tested, add tests.
-3. If you've changed APIs, update the documentation.
-4. Ensure the test suite passes.
-5. Make sure your code lints.
-6. If you haven't already, complete the Contributor License Agreement ("CLA").
-
-## Contributor License Agreement ("CLA")
-In order to accept your pull request, we need you to submit a CLA. You only need
-to do this once to work on any of Facebook's open source projects.
-
-Complete your CLA here: <https://code.facebook.com/cla>
-
-## Issues
-We use GitHub issues to track public bugs. Please ensure your description is
-clear and has sufficient instructions to be able to reproduce the issue.
-
-## Coding Style
-We follow the [PEP 8](https://www.python.org/dev/peps/pep-0008/) style guidelines.
-
-## License
-By contributing to this project, you agree that your contributions will be licensed
-under the LICENSE file in the root directory of this source tree.

VideoPose3D/DATASETS.md

@@ -1,135 +0,0 @@
-# Dataset setup
-
-## Human3.6M
-We provide two ways to set up the Human3.6M dataset on our pipeline. You can either convert the original dataset (recommended) or use the [dataset preprocessed by Martinez et al.](https://github.com/una-dinosauria/3d-pose-baseline) (no longer available as of May 22nd, 2020). The two methods produce the same result. After this step, you should end up with two files in the `data` directory: `data_3d_h36m.npz` for the 3D poses, and `data_2d_h36m_gt.npz` for the ground-truth 2D poses.
-
-### Setup from original source (recommended)
-**Update:** we have updated the instructions to simplify the procedure. MATLAB is no longer required for this step.
-
-Register to the [Human3.6m website](http://vision.imar.ro/human3.6m/) website (or login if you already have an account) and download the dataset in its original format. You only need to download *Poses -> D3 Positions* for each subject (1, 5, 6, 7, 8, 9, 11)
-
-##### Instructions without MATLAB (recommended)
-You first need to install `cdflib` Python library via `pip install cdflib`.
-
-Extract the archives named `Poses_D3_Positions_S*.tgz` (subjects 1, 5, 6, 7, 8, 9, 11) to a common directory. Your directory tree should look like this:
-
-```
-/path/to/dataset/S1/MyPoseFeatures/D3_Positions/Directions 1.cdf
-/path/to/dataset/S1/MyPoseFeatures/D3_Positions/Directions.cdf
-...
-```
-
-Then, run the preprocessing script:
-```sh
-cd data
-python prepare_data_h36m.py --from-source-cdf /path/to/dataset
-cd ..
-```
-
-If everything goes well, you are ready to go.
-
-##### Instructions with MATLAB (old instructions)
-First, we need to convert the 3D poses from `.cdf` to `.mat`, so they can be loaded from Python scripts. To this end, we have provided the MATLAB script `convert_cdf_to_mat.m` in the `data` directory. Extract the archives named `Poses_D3_Positions_S*.tgz` (subjects 1, 5, 6, 7, 8, 9, 11) to a directory named `pose`, and set up your directory tree so that it looks like this:
-
-```
-/path/to/dataset/convert_cdf_to_mat.m
-/path/to/dataset/pose/S1/MyPoseFeatures/D3_Positions/Directions 1.cdf
-/path/to/dataset/pose/S1/MyPoseFeatures/D3_Positions/Directions.cdf
-...
-```
-Then run `convert_cdf_to_mat.m` from MATLAB.
-
-Finally, run the Python conversion script specifying the dataset path:
-```sh
-cd data
-python prepare_data_h36m.py --from-source /path/to/dataset/pose
-cd ..
-```
-
-### Setup from preprocessed dataset (old instructions)
-**Update:** the link to the preprocessed dataset is no longer available; please use the procedure above. These instructions have been kept for backwards compatibility in case you already have a copy of this archive. All procedures produce the same result.
-
-Download the [~~h36m.zip archive~~](https://www.dropbox.com/s/e35qv3n6zlkouki/h36m.zip) (source: [3D pose baseline repository](https://github.com/una-dinosauria/3d-pose-baseline)) to the `data` directory, and run the conversion script from the same directory. This step does not require any additional dependency.
-
-```sh
-cd data
-wget https://www.dropbox.com/s/e35qv3n6zlkouki/h36m.zip
-python prepare_data_h36m.py --from-archive h36m.zip
-cd ..
-```
-
-## 2D detections for Human3.6M
-We provide support for the following 2D detections:
-
-- `gt`: ground-truth 2D poses, extracted through the camera projection parameters.
-- `sh_pt_mpii`: Stacked Hourglass detections (model pretrained on MPII, no fine tuning).
-- `sh_ft_h36m`: Stacked Hourglass detections, fine-tuned on Human3.6M.
-- `detectron_pt_h36m`: Detectron (Mask R-CNN) detections (model pretrained on COCO, no fine tuning).
-- `detectron_ft_h36m`: Detectron (Mask R-CNN) detections, fine-tuned on Human3.6M.
-- `cpn_ft_h36m_dbb`: Cascaded Pyramid Network detections, fine-tuned on Human3.6M. Bounding boxes from `detectron_ft_h36m`.
-- User-supplied (see below).
-
-The 2D detection source is specified through the `--keypoints` parameter, which loads the file `data_2d_DATASET_DETECTION.npz` from the `data` directory, where `DATASET` is the dataset name (e.g. `h36m`) and `DETECTION` is the 2D detection source (e.g. `sh_pt_mpii`). Since all the files are encoded according to the same format, it is trivial to create a custom set of 2D detections.
-
-Ground-truth poses (`gt`) have already been extracted by the previous step. The other detections must be downloaded manually (see instructions below). You only need to download the detections you want to use. For reference, our best results on Human3.6M are achieved by `cpn_ft_h36m_dbb`.
-
-### Mask R-CNN and CPN detections
-You can download these directly and put them in the `data` directory. We recommend starting with:
-
-```sh
-cd data
-wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_h36m_cpn_ft_h36m_dbb.npz
-wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_h36m_detectron_ft_h36m.npz
-cd ..
-```
-
-These detections have been produced by models fine-tuned on Human3.6M. We adopted the usual protocol of fine-tuning on 5 subjects (S1, S5, S6, S7, and S8). We also included detections from the unlabeled subjects S2, S3, S4, which can be loaded by our framework for semi-supervised experimentation.
-
-Optionally, you can download the Mask R-CNN detections without fine-tuning if you want to experiment with these:
-```sh
-cd data
-wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_h36m_detectron_pt_coco.npz
-cd ..
-```
-
-### Stacked Hourglass detections
-These detections (both pretrained and fine-tuned) are provided by [Martinez et al.](https://github.com/una-dinosauria/3d-pose-baseline) in their repository on 3D human pose estimation. The 2D poses produced by the pretrained model are in the same archive as the dataset ([h36m.zip](https://www.dropbox.com/s/e35qv3n6zlkouki/h36m.zip)). The fine-tuned poses can be downloaded [here](https://drive.google.com/open?id=0BxWzojlLp259S2FuUXJ6aUNxZkE). Put the two archives in the `data` directory and run:
-
-```sh
-cd data
-python prepare_data_2d_h36m_sh.py -pt h36m.zip
-python prepare_data_2d_h36m_sh.py -ft stacked_hourglass_fined_tuned_240.tar.gz
-cd ..
-```
-
-## HumanEva-I
-For HumanEva, you need the original dataset and MATLAB. We provide a MATLAB script to extract the revelant parts of the dataset automatically.
-
-1. Download the [HumanEva-I dataset](http://humaneva.is.tue.mpg.de/datasets_human_1) and extract it.
-2. Download the official [source code v1.1 beta](http://humaneva.is.tue.mpg.de/main/download?file=Release_Code_v1_1_beta.zip) and extract it where you extracted the dataset.
-3. Copy the contents of the directory `Release_Code_v1_1_beta\HumanEva_I` to the root of the source tree (`Release_Code_v1_1_beta/`).
-4. Download the [critical dataset update](http://humaneva.is.tue.mpg.de/main/download?file=Critical_Update_OFS_files.zip) and apply it.
-5. **Important:** for visualization purposes, the original code requires an old library named *dxAvi*, which is used for decoding XVID videos. A precompiled binary for 32-bit architectures is already included, but if you are running MATLAB on a 64-bit system, the code will not work. You can either recompile *dxAvi* library for x64, or bypass it entirely, since we are not using visualization features in our conversion script. To this end, you can patch `@sync_stream/sync_stream.m`, replacing line 202: `ImageStream(I) = image_stream(image_paths{I}, start_image_offset(I));` with `ImageStream(I) = 0;`
-6. Now you can copy our script `ConvertHumanEva.m` (from `data/`) to `Release_Code_v1_1_beta/`, and run it. It will create a directory named `converted_15j`, which contains the converted 2D/3D ground-truth poses on a 15-joint skeleton.
-7. **Optional:** if you want to experiment with a 20-joint skeleton, change `N_JOINTS` to 20 in `ConvertHumanEva.m`, and repeat the process. It will create a directory named `converted_20j`. Adapt next steps accordingly.
-
-If you get warnings about mocap errors or dropped frames, this is normal. The HumanEva dataset contains some invalid frames due to occlusions, which are simply discarded. Since we work with videos (and not individual frames), we try to minimize the impact of this issue by grouping valid sequences into contiguous chunks.
-
-Finally, run the Python script to produce the final files:
-```
-python prepare_data_humaneva.py -p /path/to/dataset/Release_Code_v1_1_beta/converted_15j --convert-3d
-```
-You should end up with two files in the `data` directory: `data_3d_humaneva15.npz` for the 3D poses, and `data_2d_humaneva15_gt.npz` for the ground-truth 2D poses.
-
-### 2D detections for HumanEva-I
-We provide support for the following 2D detections:
-
-- `gt`: ground-truth 2D poses, extracted through camera projection.
-- `detectron_pt_coco`: Detectron (Mask R-CNN) detections, pretrained on COCO.
-
-Since HumanEva is very small, we do not fine-tune the pretrained models. As before, you can download Mask R-CNN detections from AWS (`data_2d_humaneva15_detectron_pt_coco.npz`, which must be copied to `data/`). As before, we have included detections for unlabeled subjects/actions. These begin with the prefix `Unlabeled/`. Chunks that correspond to corrupted motion capture streams are also marked as unlabeled.
-```sh
-cd data
-wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_humaneva15_detectron_pt_coco.npz
-cd ..
-```

VideoPose3D/DOCUMENTATION.md

@@ -1,118 +0,0 @@
-# Documentation
-This guide explains in depth all the features of this framework. Make sure you have read the quick start guide in [`README.md`](README.md) before proceeding.
-
-## Training
-By default, the script `run.py` runs in training mode. The list of command-line arguments is defined in `common/arguments.py`.
-
-- `-h`: shows the help / list of parameters.
-- `-d` or `--dataset`: specifies the dataset to use (`h36m` or `humaneva15`). Default: `h36m`. If you converted the 20-joint HumanEva skeleton, you can also use `humaneva20`.
-- `-k` or `--keypoints`: specifies the 2D detections to use. Default: `cpn_ft_h36m_dbb` (CPN fine-tuned on Human 3.6M).
-- `-c` or `--checkpoint`: specifies the directory where checkpoints are saved/read. Default: `checkpoint`.
-- `--checkpoint-frequency`: save checkpoints every N epochs. Default: `10`.
-- `-r` or `--resume`: resume training from a particular checkpoint (you should only specify the file name, not the path), e.g. `epoch_10.bin`.
-- `-str` or `--subjects-train`: specifies the list of subjects on which the model is trained, separated by commas. Default: `S1,S5,S6,S7,S8`. For HumanEva, you may want to specify these manually.
-- `-ste` or `--subjects-test`: specifies the list of subjects on which the model is tested at the end of each epoch (and in the final evaluation), separated by comma. Default: `S9,S11`. For HumanEva, you may want to specify these manually.
-- `-a` or `--actions`: select only a subset of actions, separated by commas. E.g. `Walk,Jog`. By default, all actions are used.
-- `-e` or `--epochs`: train for N epochs, i.e. N passes over the entire training set. Default: `60`.
-- `--no-eval`: disable testing at the end of each epoch (marginal speed up). By default, testing is enabled.
-- `--export-training-curves`: export training curves as PNG images after every epoch. They are saved in the checkpoint directory. Default: disabled.
-
-
-If `--no-eval` is not specified, the model is tested at the end of each epoch, although the reported metric is merely an approximation of the final result (for performance reasons). Once training is over, the model is automatically tested using the full procedure. This means that you can also specify the testing parameters when training.
-
-Here is a description of the model hyperparameters:
-- `-s` or `--stride`: the chunk size used for training, i.e. the number of frames that are predicted at once from each sequence. Increasing this value improves training speed at the expense of the error (due to correlated batch statistics). Default: `1` frame, which ensures maximum decorrelation. When this value is set to `1`, we also employ an optimized implementation of the model (see implementation details).
-- `-b` or `--batch-size`: the batch size used for training the model, in terms of *output frames* (regardless of the stride/chunk length). Default: `1024` frames.
-- `-drop` or `--dropout`: dropout probability. Default: `0.25`.
-- `-lr` or `--learning-rate`: initial learning rate. Default: `0.001`.
-- `-lrd` or `--lr-decay`: learning rate decay after every epoch (multiplicative coefficient). Default: `0.95`.
-- `-no-tta` or `--no-test-time-augmentation`: disable test-time augmentation (which is enabled by default), i.e. do not flip poses horizontally when testing the model. Only effective when combined with data augmentation, so if you disable this you should also disable train-time data augmentation.
-- `-no-da` or `--no-data-augmentation`: disable train-time data augmentation (which is enabled by default), i.e. do not flip poses horizontally to double the training data.
-- `-arc` or `--architecture`: filter widths (only odd numbers supported) separated by commas. This parameter also specifies the number of residual blocks, and determines the receptive field of the model. The first number refers to the input layer, and is followed by the filter widths of the residual blocks. For instance, `3,5,5` uses `3x1` convolutions in the first layer, followed by two residual blocks with `5x1` convolutions. Default: `3,3,3`. Some valid examples are:
--- `-arc 3,3,3` (27 frames)
--- `-arc 3,3,7` (63 frames)
--- `-arc 3,3,3,3` (81 frames)
--- `-arc 3,3,3,3,3` (243 frames)
-- `--causal`: use causal (i.e. asymmetric) convolutions instead of symmetric convolutions. Causal convolutions are suitable for real-time applications because they do not exploit future frames (they only look in the past), but symmetric convolutions result in a better error since they can consider both past and future data. See below for more details.  Default: disabled.
-- `-ch` or `--channels`: number of channels in convolutions. Default: `1024`.
-- `--dense`: use dense convolutions instead of dilated convolutions. This is only useful for benchmarks and ablation experiments.
-- `--disable-optimizations`: disable the optimized implementation when `--stride` == `1`. This is only useful for benchmarks.
-
-## Semi-supervised training
-Semi-supervised learning is only implemented for Human3.6M.
-
-- `-sun` or `--subjects-unlabeled`: specifies the list of unlabeled subjects that are used for semi-supervision (separated by commas). Semi-supervised learning is automatically enabled when this parameter is set.
-- `--warmup`: number of supervised training epochs before attaching the semi-supervised loss. Default: `1` epoch. You may want to increase this when downsampling the dataset.
-- `--subset`: reduce the size of the training set by a given factor (a real number). E.g. `0.1` uses one tenth of the training data. Subsampling is achieved by extracting a random contiguous chunk from each video, while preserving the original frame rate. Default: `1` (i.e. disabled). This parameter can also be used in a supervised setting, but it is especially useful to simulate data scarcity in a semi-supervised setting.
-- `--downsample`: reduce the dataset frame rate by an integer factor. Default: `1` (i.e. disabled).
-- `--no-bone-length`: do not add the bone length term to the unsupervised loss function (only useful for ablation experiments).
-- `--linear-projection`: ignore non-linear camera distortion parameters when performing projection to 2D, i.e. use only focal length and principal point.
-- `--no-proj`: do not add the projection consistency term to the loss function (only useful for ablations).
-
-## Testing
-To test a particular model, you need to specify the checkpoint file via the `--evaluate` parameter, which will be loaded from the checkpoint directory (default: `checkpoint/`, but you can change it using the `-c` parameter). You also need to specify the same settings/hyperparameters that you used for training (e.g. input keypoints, architecture, etc.). The script will not run any compatibility checks -- this is a design choice to facilitate ablation experiments.
-
-## Visualization
-You can render videos by specifying both `--evaluate` and  `--render`. The script generates a visualization which contains three viewports: the 2D input keypoints (and optionally, a video overlay), the 3D reconstruction, and the 3D ground truth.
-Note that when you specify a video, the 2D detections are still loaded from the dataset according to the given parameters. It is up to you to choose the correct video. You can also visualize unlabeled videos -- in this case, the ground truth will not be shown.
-
-Here is a list of the command-line arguments related to visualization:
-- `--viz-subject`: subject to render, e.g. `S1`.
-- `--viz-action`: action to render, e.g. `Walking` or `Walking 1`.
-- `--viz-camera`: camera to render (integer), from 0 to 3 for Human3.6M, 0 to 2 for HumanEva. Default: `0`.
-- `--viz-video`: path to the 2D video to show. If specified, the script will render a skeleton overlay on top of the video. If not specified, a black background will be rendered instead (but the 2D detections will still be shown). 
-- `--viz-skip`: skip the first N frames from the specified video. Useful for HumanEva. Default: `0`.
-- `--viz-output`: output file name (either a `.mp4` or `.gif` file).
-- `--viz-bitrate`: bitrate for MP4 videos. Default: `3000`.
-- `--viz-no-ground-truth`: by default, the videos contain three viewports: the 2D input pose, the 3D reconstruction, and the 3D ground truth. This flags removes the last one.
-- `--viz-limit`: render only first N frames. By default, all frames are rendered.
-- `--viz-downsample`: downsample videos by the specified factor, i.e. reduce the frame rate. E.g. if set to `2`, the frame rate is reduced from 50 FPS to 25 FPS. Default: `1` (no downsampling).
-- `--viz-size`: output resolution multiplier. Higher = larger images. Default: `5`.
-- `--viz-export`: export 3D joint coordinates (in camera space) to the specified NumPy archive.
-
-Example:
-```
-python run.py -k cpn_ft_h36m_dbb -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_cpn.bin --render --viz-subject S11 --viz-action Walking --viz-camera 0 --viz-video "/path/to/videos/S11/Videos/Walking.54138969.mp4" --viz-output output.gif --viz-size 3 --viz-downsample 2 --viz-limit 60
-```
-![](images/demo_h36m.gif)
-
-Generates a visualization for S11/Walking from camera 0, and exports the first frames to a GIF animation with a frame rate of 25 FPS. If you remove the `--viz-video` parameter, the skeleton overlay will be rendered on a blank background.
-
-While Human3.6M visualization works out of the box, HumanEva visualization is trickier because the original videos must be segmented manually. Additionally, invalid frames and software synchronization complicate matters. Nonetheless, you can get decent visualizations by selecting the chunk 0 of validation sequences (which start at the beginning of each video) and discarding the first frames using `--viz-skip`. For a suggestion on the number of frames to skip, take a look at `sync_data` in `data/prepare_data_humaneva.py`.
-
-Example:
-```
-python run.py -d humaneva15 -k detectron_pt_coco -str Train/S1,Train/S2,Train/S3 -ste Validate/S1,Validate/S2,Validate/S3 -c checkpoint --evaluate pretrained_humaneva15_detectron.bin  --render --viz-subject Validate/S2 --viz-action "Walking 1 chunk0" --viz-camera 0 --viz-output output_he.gif --viz-size 3 --viz-downsample 2 --viz-video "/path/to/videos/S2/Walking_1_(C1).avi" --viz-skip 115 --viz-limit 60
-```
-![](images/demo_humaneva.gif)
-
-Unlabeled videos are easier to visualize because they do not require synchronization with the ground truth. In this case, visualization works out of the box even for HumanEva.
-
-Example:
-```
-python run.py -d humaneva15 -k detectron_pt_coco -str Train/S1,Train/S2,Train/S3 -ste Validate/S1,Validate/S2,Validate/S3 -c checkpoint --evaluate pretrained_humaneva15_detectron.bin  --render --viz-subject Unlabeled/S4 --viz-action "Box 2" --viz-camera 0 --viz-output output_he.gif --viz-size 3 --viz-downsample 2 --viz-video "/path/to/videos/S4/Box_2_(C1).avi" --viz-limit 60
-```
-![](images/demo_humaneva_unlabeled.gif)
-
-## Implementation details
-### Batch generation during training
-Some details of our training procedure are better understood visually.
-![](images/batching.png)
-The figure above shows how training batches are generated, depending on the value of `--stride` (from left to right: 1, 2, and 4). This example shows a sequence of 2D poses which has a length of N = 8 frames. The 3D poses (blue boxes in the figure) are inferred using a model that has a receptive field F = 5 frames. Therefore, because of valid padding, an input sequence of length N results in an output sequence of length N - F + 1, i.e. N - 4 in this example.
-
-When `--stride=1`, we generate one training example for each frame. This ensures that the batches are maximally uncorrelated, which helps batch normalization as well as generalization. As `--stride` increases, training becomes faster because the model can reutilize intermediate computations, at the cost of biased batch statistics. However, we provide an optimized implementation when `--stride=1`, which replaces dilated convolutions with strided convolutions (only while training), so in principle you should not touch this parameter unless you want to run specific experiments. To understand how it works, see the figures below:
-
-![](images/convolutions_1f_naive.png)
-The figure above shows the information flow for a model with a receptive field of 27 frames, and a single-frame prediction, i.e. from N = 27 input frames we end up with one output frame. You can observe that this regular implementation tends to waste some intermediate results when a small number of frames are predicted. However, for inference of very long sequences, this approach is very efficient as intermediate results are shared among successive frames.
-
-![](images/convolutions_1f_optimized.png)
-Therefore, for training *only*, we use the implementation above, which replaces dilated convolutions with strided convolutions. It achieves the same result, but avoids computing unnecessary intermediate results.
-
-### Symmetric convolutions vs causal convolutions
-The figures below show the information flow from input (bottom) to output (top). In this example, we adopt a model with a receptive field of 27 frames.
-
-![](images/convolutions_normal.png)
-With symmetric convolutions, both past and future information is exploited, resulting in a better reconstruction.
-
-![](images/convolutions_causal.png)
-With causal convolutions, only past data is exploited. This approach is suited to real-time applications where future data cannot be exploited, at the cost of a slightly higher error.

VideoPose3D/INFERENCE.md

@@ -1,85 +0,0 @@
-# Inference in the wild
-
-**Update:** we have added support for Detectron2.
-
-In this short tutorial, we show how to run our model on arbitrary videos and visualize the predictions. Note that this feature is only provided for experimentation/research purposes and presents some limitations, as this repository is meant to provide a reference implementation of the approach described in the paper (not production-ready code for inference in the wild).
-
-Our script assumes that a video depicts *exactly* one person. In case of multiple people visible at once, the script will select the person corresponding to the bounding box with the highest confidence, which may cause glitches.
-
-The instructions below show how to use Detectron to infer 2D keypoints from videos, convert them to a custom dataset for our code, and infer 3D poses. For now, we do not have instructions for CPN. In the last section of this tutorial, we also provide some tips.
-
-## Step 1: setup
-The inference script requires `ffmpeg`, which you can easily install via conda, pip, or manually.
-
-Download the [pretrained model](https://dl.fbaipublicfiles.com/video-pose-3d/pretrained_h36m_detectron_coco.bin) for generating 3D predictions. This model is different than the pretrained ones listed in the main README, as it expects input keypoints in COCO format (generated by the pretrained Detectron model) and outputs 3D joint positions in Human3.6M format. Put this model in the `checkpoint` directory of this repo.
-
-**Note:** if you had downloaded `d-pt-243.bin`, you should download the new pretrained model using the link above. `d-pt-243.bin` takes the keypoint probabilities as input (in addition to the x, y coordinates), which causes problems on videos with a different resolution than that of Human3.6M. The new model is only trained on 2D coordinates and works with any resolution/aspect ratio.
-
-## Step 2 (optional): video preprocessing
-Since the script expects a single-person scenario, you may want to extract a portion of your video. This is very easy to do with ffmpeg, e.g.
-```
-ffmpeg -i input.mp4 -ss 1:00 -to 1:30 -c copy output.mp4
-```
-extracts a clip from minute 1:00 to minute 1:30 of `input.mp4`, and exports it to `output.mp4`.
-
-Optionally, you can also adapt the frame rate of the video. Most videos have a frame rate of about 25 FPS, but our Human3.6M model was trained on 50-FPS videos. Since our model is robust to alterations in speed, this step is not very important and can be skipped, but if you want the best possible results you can use ffmpeg again for this task:
-```
-ffmpeg -i input.mp4 -filter "minterpolate='fps=50'" -crf 0 output.mp4
-```
-
-## Step 3: inferring 2D keypoints with Detectron
-
-### Using Detectron2 (new)
-Set up [Detectron2](https://github.com/facebookresearch/detectron2) and use the script  `inference/infer_video_d2.py` (no need to copy this, as it directly uses the Detectron2 API). This script provides a convenient interface to generate 2D keypoint predictions from videos without manually extracting individual frames.
-
-To infer keypoints from all the mp4 videos in `input_directory`, run
-```
-cd inference
-python infer_video_d2.py \
-    --cfg COCO-Keypoints/keypoint_rcnn_R_101_FPN_3x.yaml \
-    --output-dir output_directory \
-    --image-ext mp4 \
-    input_directory
-```
-The results will be exported to `output_directory` as custom NumPy archives (`.npz` files). You can change the video extension in `--image-ext` (ffmpeg supports a wide range of formats).
-
-**Note:** although the architecture is the same (ResNet-101), the weights used by the Detectron2 model are not the same as those used by Detectron1. Since our pretrained model was trained on Detectron1 poses, the result might be slightly different (but it should still be pretty close).
-
-### Using Detectron1 (old instructions)
-Set up [Detectron](https://github.com/facebookresearch/Detectron) and copy the script `inference/infer_video.py` from this repo to the `tools` directory of the Detectron repo. This script provides a convenient interface to generate 2D keypoint predictions from videos without manually extracting individual frames.
-
-Our Detectron script `infer_video.py` is a simple adaptation of `infer_simple.py` (which works on images) and has a similar command-line syntax.
-
-To infer keypoints from all the mp4 videos in `input_directory`, run
-```
-python tools/infer_video.py \
-    --cfg configs/12_2017_baselines/e2e_keypoint_rcnn_R-101-FPN_s1x.yaml \
-    --output-dir output_directory \
-    --image-ext mp4 \
-	--wts https://dl.fbaipublicfiles.com/detectron/37698009/12_2017_baselines/e2e_keypoint_rcnn_R-101-FPN_s1x.yaml.08_45_57.YkrJgP6O/output/train/keypoints_coco_2014_train:keypoints_coco_2014_valminusminival/generalized_rcnn/model_final.pkl \
-    input_directory
-```
-The results will be exported to `output_directory` as custom NumPy archives (`.npz` files). You can change the video extension in `--image-ext` (ffmpeg supports a wide range of formats).
-
-## Step 4: creating a custom dataset
-Run our dataset preprocessing script from the `data` directory:
-```
-python prepare_data_2d_custom.py -i /path/to/detections/output_directory -o myvideos
-```
-This creates a custom dataset named `myvideos` (which contains all the videos in `output_directory`, each of which is mapped to a different subject) and saved to `data_2d_custom_myvideos.npz`. You are free to specify any name for the dataset.
-
-**Note:** as mentioned, the script will take the bounding box with the highest probability for each frame. If a particular frame has no bounding boxes, it is assumed to be a missed detection and the keypoints will be interpolated from neighboring frames.
-
-## Step 5: rendering a custom video and exporting coordinates
-You can finally use the visualization feature to render a video of the 3D joint predictions. You must specify the `custom` dataset (`-d custom`), the input keypoints as exported in the previous step (`-k myvideos`), the correct architecture/checkpoint, and the action `custom` (`--viz-action custom`). The subject is the file name of the input video, and the camera is always 0.
-```
-python run.py -d custom -k myvideos -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_detectron_coco.bin --render --viz-subject input_video.mp4 --viz-action custom --viz-camera 0 --viz-video /path/to/input_video.mp4 --viz-output output.mp4 --viz-size 6
-```
-
-You can also export the 3D joint positions (in camera space) to a NumPy archive. To this end, replace `--viz-output` with `--viz-export` and specify the file name.
-
-## Limitations and tips
-- The model was trained on Human3.6M cameras (which are relatively undistorted), and the results may be bad if the intrinsic parameters of the cameras of your videos differ much from those of Human3.6M. This may be particularly noticeable with fisheye cameras, which present a high degree of non-linear lens distortion. If the camera parameters are known, consider preprocessing your videos to match those of Human3.6M as closely as possible.
-- If you want multi-person tracking, you should implement a bounding box matching strategy. An example would be to use bipartite matching on the bounding box overlap (IoU) between subsequent frames, but there are many other approaches.
-- Predictions are relative to the root joint, i.e. the global trajectory is not regressed. If you need it, you may want to use another model to regress it, such as the one we use for semi-supervision.
-- Predictions are always in *camera space* (regardless of whether the trajectory is available). For our visualization script, we simply take a random camera from Human3.6M, which fits decently most videos where the camera viewport is parallel to the ground. 

VideoPose3D/LICENSE

@@ -1,399 +0,0 @@
-Attribution-NonCommercial 4.0 International
-
-=======================================================================
-
-Creative Commons Corporation ("Creative Commons") is not a law firm and
-does not provide legal services or legal advice. Distribution of
-Creative Commons public licenses does not create a lawyer-client or
-other relationship. Creative Commons makes its licenses and related
-information available on an "as-is" basis. Creative Commons gives no
-warranties regarding its licenses, any material licensed under their
-terms and conditions, or any related information. Creative Commons
-disclaims all liability for damages resulting from their use to the
-fullest extent possible.
-
-Using Creative Commons Public Licenses
-
-Creative Commons public licenses provide a standard set of terms and
-conditions that creators and other rights holders may use to share
-original works of authorship and other material subject to copyright
-and certain other rights specified in the public license below. The
-following considerations are for informational purposes only, are not
-exhaustive, and do not form part of our licenses.
-
-     Considerations for licensors: Our public licenses are
-     intended for use by those authorized to give the public
-     permission to use material in ways otherwise restricted by
-     copyright and certain other rights. Our licenses are
-     irrevocable. Licensors should read and understand the terms
-     and conditions of the license they choose before applying it.
-     Licensors should also secure all rights necessary before
-     applying our licenses so that the public can reuse the
-     material as expected. Licensors should clearly mark any
-     material not subject to the license. This includes other CC-
-     licensed material, or material used under an exception or
-     limitation to copyright. More considerations for licensors:
-     wiki.creativecommons.org/Considerations_for_licensors
-
-     Considerations for the public: By using one of our public
-     licenses, a licensor grants the public permission to use the
-     licensed material under specified terms and conditions. If
-     the licensor's permission is not necessary for any reason--for
-     example, because of any applicable exception or limitation to
-     copyright--then that use is not regulated by the license. Our
-     licenses grant only permissions under copyright and certain
-     other rights that a licensor has authority to grant. Use of
-     the licensed material may still be restricted for other
-     reasons, including because others have copyright or other
-     rights in the material. A licensor may make special requests,
-     such as asking that all changes be marked or described.
-     Although not required by our licenses, you are encouraged to
-     respect those requests where reasonable. More_considerations
-     for the public: 
-     wiki.creativecommons.org/Considerations_for_licensees
-
-=======================================================================
-
-Creative Commons Attribution-NonCommercial 4.0 International Public
-License
-
-By exercising the Licensed Rights (defined below), You accept and agree
-to be bound by the terms and conditions of this Creative Commons
-Attribution-NonCommercial 4.0 International Public License ("Public
-License"). To the extent this Public License may be interpreted as a
-contract, You are granted the Licensed Rights in consideration of Your
-acceptance of these terms and conditions, and the Licensor grants You
-such rights in consideration of benefits the Licensor receives from
-making the Licensed Material available under these terms and
-conditions.
-
-Section 1 -- Definitions.
-
-  a. Adapted Material means material subject to Copyright and Similar
-     Rights that is derived from or based upon the Licensed Material
-     and in which the Licensed Material is translated, altered,
-     arranged, transformed, or otherwise modified in a manner requiring
-     permission under the Copyright and Similar Rights held by the
-     Licensor. For purposes of this Public License, where the Licensed
-     Material is a musical work, performance, or sound recording,
-     Adapted Material is always produced where the Licensed Material is
-     synched in timed relation with a moving image.
-
-  b. Adapter's License means the license You apply to Your Copyright
-     and Similar Rights in Your contributions to Adapted Material in
-     accordance with the terms and conditions of this Public License.
-
-  c. Copyright and Similar Rights means copyright and/or similar rights
-     closely related to copyright including, without limitation,
-     performance, broadcast, sound recording, and Sui Generis Database
-     Rights, without regard to how the rights are labeled or
-     categorized. For purposes of this Public License, the rights
-     specified in Section 2(b)(1)-(2) are not Copyright and Similar
-     Rights.
-  d. Effective Technological Measures means those measures that, in the
-     absence of proper authority, may not be circumvented under laws
-     fulfilling obligations under Article 11 of the WIPO Copyright
-     Treaty adopted on December 20, 1996, and/or similar international
-     agreements.
-
-  e. Exceptions and Limitations means fair use, fair dealing, and/or
-     any other exception or limitation to Copyright and Similar Rights
-     that applies to Your use of the Licensed Material.
-
-  f. Licensed Material means the artistic or literary work, database,
-     or other material to which the Licensor applied this Public
-     License.
-
-  g. Licensed Rights means the rights granted to You subject to the
-     terms and conditions of this Public License, which are limited to
-     all Copyright and Similar Rights that apply to Your use of the
-     Licensed Material and that the Licensor has authority to license.
-
-  h. Licensor means the individual(s) or entity(ies) granting rights
-     under this Public License.
-
-  i. NonCommercial means not primarily intended for or directed towards
-     commercial advantage or monetary compensation. For purposes of
-     this Public License, the exchange of the Licensed Material for
-     other material subject to Copyright and Similar Rights by digital
-     file-sharing or similar means is NonCommercial provided there is
-     no payment of monetary compensation in connection with the
-     exchange.
-
-  j. Share means to provide material to the public by any means or
-     process that requires permission under the Licensed Rights, such
-     as reproduction, public display, public performance, distribution,
-     dissemination, communication, or importation, and to make material
-     available to the public including in ways that members of the
-     public may access the material from a place and at a time
-     individually chosen by them.
-
-  k. Sui Generis Database Rights means rights other than copyright
-     resulting from Directive 96/9/EC of the European Parliament and of
-     the Council of 11 March 1996 on the legal protection of databases,
-     as amended and/or succeeded, as well as other essentially
-     equivalent rights anywhere in the world.
-
-  l. You means the individual or entity exercising the Licensed Rights
-     under this Public License. Your has a corresponding meaning.
-
-Section 2 -- Scope.
-
-  a. License grant.
-
-       1. Subject to the terms and conditions of this Public License,
-          the Licensor hereby grants You a worldwide, royalty-free,
-          non-sublicensable, non-exclusive, irrevocable license to
-          exercise the Licensed Rights in the Licensed Material to:
-
-            a. reproduce and Share the Licensed Material, in whole or
-               in part, for NonCommercial purposes only; and
-
-            b. produce, reproduce, and Share Adapted Material for
-               NonCommercial purposes only.
-
-       2. Exceptions and Limitations. For the avoidance of doubt, where
-          Exceptions and Limitations apply to Your use, this Public
-          License does not apply, and You do not need to comply with
-          its terms and conditions.
-
-       3. Term. The term of this Public License is specified in Section
-          6(a).
-
-       4. Media and formats; technical modifications allowed. The
-          Licensor authorizes You to exercise the Licensed Rights in
-          all media and formats whether now known or hereafter created,
-          and to make technical modifications necessary to do so. The
-          Licensor waives and/or agrees not to assert any right or
-          authority to forbid You from making technical modifications
-          necessary to exercise the Licensed Rights, including
-          technical modifications necessary to circumvent Effective
-          Technological Measures. For purposes of this Public License,
-          simply making modifications authorized by this Section 2(a)
-          (4) never produces Adapted Material.
-
-       5. Downstream recipients.
-
-            a. Offer from the Licensor -- Licensed Material. Every
-               recipient of the Licensed Material automatically
-               receives an offer from the Licensor to exercise the
-               Licensed Rights under the terms and conditions of this
-               Public License.
-
-            b. No downstream restrictions. You may not offer or impose
-               any additional or different terms or conditions on, or
-               apply any Effective Technological Measures to, the
-               Licensed Material if doing so restricts exercise of the
-               Licensed Rights by any recipient of the Licensed
-               Material.
-
-       6. No endorsement. Nothing in this Public License constitutes or
-          may be construed as permission to assert or imply that You
-          are, or that Your use of the Licensed Material is, connected
-          with, or sponsored, endorsed, or granted official status by,
-          the Licensor or others designated to receive attribution as
-          provided in Section 3(a)(1)(A)(i).
-
-  b. Other rights.
-
-       1. Moral rights, such as the right of integrity, are not
-          licensed under this Public License, nor are publicity,
-          privacy, and/or other similar personality rights; however, to
-          the extent possible, the Licensor waives and/or agrees not to
-          assert any such rights held by the Licensor to the limited
-          extent necessary to allow You to exercise the Licensed
-          Rights, but not otherwise.
-
-       2. Patent and trademark rights are not licensed under this
-          Public License.
-
-       3. To the extent possible, the Licensor waives any right to
-          collect royalties from You for the exercise of the Licensed
-          Rights, whether directly or through a collecting society
-          under any voluntary or waivable statutory or compulsory
-          licensing scheme. In all other cases the Licensor expressly
-          reserves any right to collect such royalties, including when
-          the Licensed Material is used other than for NonCommercial
-          purposes.
-
-Section 3 -- License Conditions.
-
-Your exercise of the Licensed Rights is expressly made subject to the
-following conditions.
-
-  a. Attribution.
-
-       1. If You Share the Licensed Material (including in modified
-          form), You must:
-
-            a. retain the following if it is supplied by the Licensor
-               with the Licensed Material:
-
-                 i. identification of the creator(s) of the Licensed
-                    Material and any others designated to receive
-                    attribution, in any reasonable manner requested by
-                    the Licensor (including by pseudonym if
-                    designated);
-
-                ii. a copyright notice;
-
-               iii. a notice that refers to this Public License;
-
-                iv. a notice that refers to the disclaimer of
-                    warranties;
-
-                 v. a URI or hyperlink to the Licensed Material to the
-                    extent reasonably practicable;
-
-            b. indicate if You modified the Licensed Material and
-               retain an indication of any previous modifications; and
-
-            c. indicate the Licensed Material is licensed under this
-               Public License, and include the text of, or the URI or
-               hyperlink to, this Public License.
-
-       2. You may satisfy the conditions in Section 3(a)(1) in any
-          reasonable manner based on the medium, means, and context in
-          which You Share the Licensed Material. For example, it may be
-          reasonable to satisfy the conditions by providing a URI or
-          hyperlink to a resource that includes the required
-          information.
-
-       3. If requested by the Licensor, You must remove any of the
-          information required by Section 3(a)(1)(A) to the extent
-          reasonably practicable.
-
-       4. If You Share Adapted Material You produce, the Adapter's
-          License You apply must not prevent recipients of the Adapted
-          Material from complying with this Public License.
-
-Section 4 -- Sui Generis Database Rights.
-
-Where the Licensed Rights include Sui Generis Database Rights that
-apply to Your use of the Licensed Material:
-
-  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
-     to extract, reuse, reproduce, and Share all or a substantial
-     portion of the contents of the database for NonCommercial purposes
-     only;
-
-  b. if You include all or a substantial portion of the database
-     contents in a database in which You have Sui Generis Database
-     Rights, then the database in which You have Sui Generis Database
-     Rights (but not its individual contents) is Adapted Material; and
-
-  c. You must comply with the conditions in Section 3(a) if You Share
-     all or a substantial portion of the contents of the database.
-
-For the avoidance of doubt, this Section 4 supplements and does not
-replace Your obligations under this Public License where the Licensed
-Rights include other Copyright and Similar Rights.
-
-Section 5 -- Disclaimer of Warranties and Limitation of Liability.
-
-  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
-     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
-     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
-     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
-     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
-     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
-     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
-     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
-     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
-     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
-
-  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
-     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
-     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
-     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
-     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
-     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
-     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
-     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
-     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
-
-  c. The disclaimer of warranties and limitation of liability provided
-     above shall be interpreted in a manner that, to the extent
-     possible, most closely approximates an absolute disclaimer and
-     waiver of all liability.
-
-Section 6 -- Term and Termination.
-
-  a. This Public License applies for the term of the Copyright and
-     Similar Rights licensed here. However, if You fail to comply with
-     this Public License, then Your rights under this Public License
-     terminate automatically.
-
-  b. Where Your right to use the Licensed Material has terminated under
-     Section 6(a), it reinstates:
-
-       1. automatically as of the date the violation is cured, provided
-          it is cured within 30 days of Your discovery of the
-          violation; or
-
-       2. upon express reinstatement by the Licensor.
-
-     For the avoidance of doubt, this Section 6(b) does not affect any
-     right the Licensor may have to seek remedies for Your violations
-     of this Public License.
-
-  c. For the avoidance of doubt, the Licensor may also offer the
-     Licensed Material under separate terms or conditions or stop
-     distributing the Licensed Material at any time; however, doing so
-     will not terminate this Public License.
-
-  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
-     License.
-
-Section 7 -- Other Terms and Conditions.
-
-  a. The Licensor shall not be bound by any additional or different
-     terms or conditions communicated by You unless expressly agreed.
-
-  b. Any arrangements, understandings, or agreements regarding the
-     Licensed Material not stated herein are separate from and
-     independent of the terms and conditions of this Public License.
-
-Section 8 -- Interpretation.
-
-  a. For the avoidance of doubt, this Public License does not, and
-     shall not be interpreted to, reduce, limit, restrict, or impose
-     conditions on any use of the Licensed Material that could lawfully
-     be made without permission under this Public License.
-
-  b. To the extent possible, if any provision of this Public License is
-     deemed unenforceable, it shall be automatically reformed to the
-     minimum extent necessary to make it enforceable. If the provision
-     cannot be reformed, it shall be severed from this Public License
-     without affecting the enforceability of the remaining terms and
-     conditions.
-
-  c. No term or condition of this Public License will be waived and no
-     failure to comply consented to unless expressly agreed to by the
-     Licensor.
-
-  d. Nothing in this Public License constitutes or may be interpreted
-     as a limitation upon, or waiver of, any privileges and immunities
-     that apply to the Licensor or You, including from the legal
-     processes of any jurisdiction or authority.
-
-=======================================================================
-
-Creative Commons is not a party to its public
-licenses. Notwithstanding, Creative Commons may elect to apply one of
-its public licenses to material it publishes and in those instances
-will be considered the “Licensor.” The text of the Creative Commons
-public licenses is dedicated to the public domain under the CC0 Public
-Domain Dedication. Except for the limited purpose of indicating that
-material is shared under a Creative Commons public license or as
-otherwise permitted by the Creative Commons policies published at
-creativecommons.org/policies, Creative Commons does not authorize the
-use of the trademark "Creative Commons" or any other trademark or logo
-of Creative Commons without its prior written consent including,
-without limitation, in connection with any unauthorized modifications
-to any of its public licenses or any other arrangements,
-understandings, or agreements concerning use of licensed material. For
-the avoidance of doubt, this paragraph does not form part of the
-public licenses.
-
-Creative Commons may be contacted at creativecommons.org.

VideoPose3D/README.md

@@ -1,119 +0,0 @@
-# 3D human pose estimation in video with temporal convolutions and semi-supervised training
-<p align="center"><img src="images/convolutions_anim.gif" width="50%" alt="" /></p>
-
-This is the implementation of the approach described in the paper:
-> Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://arxiv.org/abs/1811.11742). In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
-
-More demos are available at https://dariopavllo.github.io/VideoPose3D
-
-<p align="center"><img src="images/demo_yt.gif" width="70%" alt="" /></p>
-
-![](images/demo_temporal.gif)
-
-### Results on Human3.6M
-Under Protocol 1 (mean per-joint position error) and Protocol 2 (mean-per-joint position error after rigid alignment).
-
-| 2D Detections | BBoxes | Blocks | Receptive Field | Error (P1) | Error (P2) |
-|:-------|:-------:|:-------:|:-------:|:-------:|:-------:|
-| CPN | Mask R-CNN  | 4 | 243 frames | **46.8 mm** | **36.5 mm** |
-| CPN | Ground truth | 4 | 243 frames | 47.1 mm | 36.8 mm |
-| CPN | Ground truth | 3 | 81 frames | 47.7 mm | 37.2 mm |
-| CPN | Ground truth | 2 | 27 frames | 48.8 mm | 38.0 mm |
-| Mask R-CNN | Mask R-CNN | 4 | 243 frames | 51.6 mm | 40.3 mm |
-| Ground truth | -- | 4 | 243 frames | 37.2 mm | 27.2 mm |
-
-## Quick start
-To get started as quickly as possible, follow the instructions in this section. This should allow you train a model from scratch, test our pretrained models, and produce basic visualizations. For more detailed instructions, please refer to [`DOCUMENTATION.md`](DOCUMENTATION.md).
-
-### Dependencies
-Make sure you have the following dependencies installed before proceeding:
-- Python 3+ distribution
-- PyTorch >= 0.4.0
-
-Optional:
-- Matplotlib, if you want to visualize predictions. Additionally, you need *ffmpeg* to export MP4 videos, and *imagemagick* to export GIFs.
-- MATLAB, if you want to experiment with HumanEva-I (you need this to convert the dataset). 
-
-### Dataset setup
-You can find the instructions for setting up the Human3.6M and HumanEva-I datasets in [`DATASETS.md`](DATASETS.md). For this short guide, we focus on Human3.6M. You are not required to setup HumanEva, unless you want to experiment with it.
-
-In order to proceed, you must also copy CPN detections (for Human3.6M) and/or Mask R-CNN detections (for HumanEva).
-
-### Evaluating our pretrained models
-The pretrained models can be downloaded from AWS. Put `pretrained_h36m_cpn.bin` (for Human3.6M) and/or `pretrained_humaneva15_detectron.bin` (for HumanEva) in the `checkpoint/` directory (create it if it does not exist).
-```sh
-mkdir checkpoint
-cd checkpoint
-wget https://dl.fbaipublicfiles.com/video-pose-3d/pretrained_h36m_cpn.bin
-wget https://dl.fbaipublicfiles.com/video-pose-3d/pretrained_humaneva15_detectron.bin
-cd ..
-```
-
-These models allow you to reproduce our top-performing baselines, which are:
-- 46.8 mm for Human3.6M, using fine-tuned CPN detections, bounding boxes from Mask R-CNN, and an architecture with a receptive field of 243 frames.
-- 33.0 mm for HumanEva-I (on 3 actions), using pretrained Mask R-CNN detections, and an architecture with a receptive field of 27 frames. This is the multi-action model trained on 3 actions (Walk, Jog, Box).
-
-To test on Human3.6M, run:
-```
-python run.py -k cpn_ft_h36m_dbb -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_cpn.bin
-```
-
-To test on HumanEva, run:
-```
-python run.py -d humaneva15 -k detectron_pt_coco -str Train/S1,Train/S2,Train/S3 -ste Validate/S1,Validate/S2,Validate/S3 -a Walk,Jog,Box --by-subject -c checkpoint --evaluate pretrained_humaneva15_detectron.bin
-```
-
-[`DOCUMENTATION.md`](DOCUMENTATION.md) provides a precise description of all command-line arguments.
-
-### Inference in the wild
-We have introduced an experimental feature to run our model on custom videos. See [`INFERENCE.md`](INFERENCE.md) for more details.
-
-### Training from scratch
-If you want to reproduce the results of our pretrained models, run the following commands.
-
-For Human3.6M:
-```
-python run.py -e 80 -k cpn_ft_h36m_dbb -arc 3,3,3,3,3
-```
-By default the application runs in training mode. This will train a new model for 80 epochs, using fine-tuned CPN detections. Expect a training time of 24 hours on a high-end Pascal GPU. If you feel that this is too much, or your GPU is not powerful enough, you can train a model with a smaller receptive field, e.g.
-- `-arc 3,3,3,3` (81 frames) should require 11 hours and achieve 47.7 mm. 
-- `-arc 3,3,3` (27 frames) should require 6 hours and achieve 48.8 mm.
-
-You could also lower the number of epochs from 80 to 60 with a negligible impact on the result.
-
-For HumanEva:
-```
-python run.py -d humaneva15 -k detectron_pt_coco -str Train/S1,Train/S2,Train/S3 -ste Validate/S1,Validate/S2,Validate/S3 -b 128 -e 1000 -lrd 0.996 -a Walk,Jog,Box --by-subject
-```
-This will train for 1000 epochs, using Mask R-CNN detections and evaluating each subject separately.
-Since HumanEva is much smaller than Human3.6M, training should require about 50 minutes.
-
-### Semi-supervised training
-To perform semi-supervised training, you just need to add the `--subjects-unlabeled` argument. In the example below, we use ground-truth 2D poses as input, and train supervised on just 10% of Subject 1 (specified by `--subset 0.1`). The remaining subjects are treated as unlabeled data and are used for semi-supervision.
-```
-python run.py -k gt --subjects-train S1 --subset 0.1 --subjects-unlabeled S5,S6,S7,S8 -e 200 -lrd 0.98 -arc 3,3,3 --warmup 5 -b 64
-```
-This should give you an error around 65.2 mm. By contrast, if we only train supervised
-```
-python run.py -k gt --subjects-train S1 --subset 0.1 -e 200 -lrd 0.98 -arc 3,3,3 -b 64
-```
-we get around 80.7 mm, which is significantly higher.
-
-### Visualization
-If you have the original Human3.6M videos, you can generate nice visualizations of the model predictions. For instance:
-```
-python run.py -k cpn_ft_h36m_dbb -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_cpn.bin --render --viz-subject S11 --viz-action Walking --viz-camera 0 --viz-video "/path/to/videos/S11/Videos/Walking.54138969.mp4" --viz-output output.gif --viz-size 3 --viz-downsample 2 --viz-limit 60
-```
-The script can also export MP4 videos, and supports a variety of parameters (e.g. downsampling/FPS, size, bitrate). See [`DOCUMENTATION.md`](DOCUMENTATION.md) for more details.
-
-## License
-This work is licensed under CC BY-NC. See LICENSE for details. Third-party datasets are subject to their respective licenses.
-If you use our code/models in your research, please cite our paper:
-```
-@inproceedings{pavllo:videopose3d:2019,
-  title={3D human pose estimation in video with temporal convolutions and semi-supervised training},
-  author={Pavllo, Dario and Feichtenhofer, Christoph and Grangier, David and Auli, Michael},
-  booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
-  year={2019}
-}
-```

VideoPose3D/common/arguments.py

@@ -1,89 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import argparse
-
-def parse_args():
-    parser = argparse.ArgumentParser(description='Training script')
-
-    # General arguments
-    parser.add_argument('-d', '--dataset', default='h36m', type=str, metavar='NAME', help='target dataset') # h36m or humaneva
-    parser.add_argument('-k', '--keypoints', default='cpn_ft_h36m_dbb', type=str, metavar='NAME', help='2D detections to use')
-    parser.add_argument('-str', '--subjects-train', default='S1,S5,S6,S7,S8', type=str, metavar='LIST',
-                        help='training subjects separated by comma')
-    parser.add_argument('-ste', '--subjects-test', default='S9,S11', type=str, metavar='LIST', help='test subjects separated by comma')
-    parser.add_argument('-sun', '--subjects-unlabeled', default='', type=str, metavar='LIST',
-                        help='unlabeled subjects separated by comma for self-supervision')
-    parser.add_argument('-a', '--actions', default='*', type=str, metavar='LIST',
-                        help='actions to train/test on, separated by comma, or * for all')
-    parser.add_argument('-c', '--checkpoint', default='checkpoint', type=str, metavar='PATH',
-                        help='checkpoint directory')
-    parser.add_argument('--checkpoint-frequency', default=10, type=int, metavar='N',
-                        help='create a checkpoint every N epochs')
-    parser.add_argument('-r', '--resume', default='', type=str, metavar='FILENAME',
-                        help='checkpoint to resume (file name)')
-    parser.add_argument('--evaluate', default='', type=str, metavar='FILENAME', help='checkpoint to evaluate (file name)')
-    parser.add_argument('--render', action='store_true', help='visualize a particular video')
-    parser.add_argument('--by-subject', action='store_true', help='break down error by subject (on evaluation)')
-    parser.add_argument('--export-training-curves', action='store_true', help='save training curves as .png images')
-
-    # Model arguments
-    parser.add_argument('-s', '--stride', default=1, type=int, metavar='N', help='chunk size to use during training')
-    parser.add_argument('-e', '--epochs', default=60, type=int, metavar='N', help='number of training epochs')
-    parser.add_argument('-b', '--batch-size', default=1024, type=int, metavar='N', help='batch size in terms of predicted frames')
-    parser.add_argument('-drop', '--dropout', default=0.25, type=float, metavar='P', help='dropout probability')
-    parser.add_argument('-lr', '--learning-rate', default=0.001, type=float, metavar='LR', help='initial learning rate')
-    parser.add_argument('-lrd', '--lr-decay', default=0.95, type=float, metavar='LR', help='learning rate decay per epoch')
-    parser.add_argument('-no-da', '--no-data-augmentation', dest='data_augmentation', action='store_false',
-                        help='disable train-time flipping')
-    parser.add_argument('-no-tta', '--no-test-time-augmentation', dest='test_time_augmentation', action='store_false',
-                        help='disable test-time flipping')
-    parser.add_argument('-arc', '--architecture', default='3,3,3', type=str, metavar='LAYERS', help='filter widths separated by comma')
-    parser.add_argument('--causal', action='store_true', help='use causal convolutions for real-time processing')
-    parser.add_argument('-ch', '--channels', default=1024, type=int, metavar='N', help='number of channels in convolution layers')
-
-    # Experimental
-    parser.add_argument('--subset', default=1, type=float, metavar='FRACTION', help='reduce dataset size by fraction')
-    parser.add_argument('--downsample', default=1, type=int, metavar='FACTOR', help='downsample frame rate by factor (semi-supervised)')
-    parser.add_argument('--warmup', default=1, type=int, metavar='N', help='warm-up epochs for semi-supervision')
-    parser.add_argument('--no-eval', action='store_true', help='disable epoch evaluation while training (small speed-up)')
-    parser.add_argument('--dense', action='store_true', help='use dense convolutions instead of dilated convolutions')
-    parser.add_argument('--disable-optimizations', action='store_true', help='disable optimized model for single-frame predictions')
-    parser.add_argument('--linear-projection', action='store_true', help='use only linear coefficients for semi-supervised projection')
-    parser.add_argument('--no-bone-length', action='store_false', dest='bone_length_term',
-                        help='disable bone length term in semi-supervised settings')
-    parser.add_argument('--no-proj', action='store_true', help='disable projection for semi-supervised setting')
-    
-    # Visualization
-    parser.add_argument('--viz-subject', type=str, metavar='STR', help='subject to render')
-    parser.add_argument('--viz-action', type=str, metavar='STR', help='action to render')
-    parser.add_argument('--viz-camera', type=int, default=0, metavar='N', help='camera to render')
-    parser.add_argument('--viz-video', type=str, metavar='PATH', help='path to input video')
-    parser.add_argument('--viz-skip', type=int, default=0, metavar='N', help='skip first N frames of input video')
-    parser.add_argument('--viz-output', type=str, metavar='PATH', help='output file name (.gif or .mp4)')
-    parser.add_argument('--viz-export', type=str, metavar='PATH', help='output file name for coordinates')
-    parser.add_argument('--viz-bitrate', type=int, default=3000, metavar='N', help='bitrate for mp4 videos')
-    parser.add_argument('--viz-no-ground-truth', action='store_true', help='do not show ground-truth poses')
-    parser.add_argument('--viz-limit', type=int, default=-1, metavar='N', help='only render first N frames')
-    parser.add_argument('--viz-downsample', type=int, default=1, metavar='N', help='downsample FPS by a factor N')
-    parser.add_argument('--viz-size', type=int, default=5, metavar='N', help='image size')
-    
-    parser.set_defaults(bone_length_term=True)
-    parser.set_defaults(data_augmentation=True)
-    parser.set_defaults(test_time_augmentation=True)
-    
-    args = parser.parse_args()
-    # Check invalid configuration
-    if args.resume and args.evaluate:
-        print('Invalid flags: --resume and --evaluate cannot be set at the same time')
-        exit()
-        
-    if args.export_training_curves and args.no_eval:
-        print('Invalid flags: --export-training-curves and --no-eval cannot be set at the same time')
-        exit()
-
-    return args

VideoPose3D/common/camera.py

@@ -1,90 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-import torch
-
-from common.utils import wrap
-from common.quaternion import qrot, qinverse
-
-def normalize_screen_coordinates(X, w, h): 
-    assert X.shape[-1] == 2
-    
-    # Normalize so that [0, w] is mapped to [-1, 1], while preserving the aspect ratio
-    return X/w*2 - [1, h/w]
-
-    
-def image_coordinates(X, w, h):
-    assert X.shape[-1] == 2
-    
-    # Reverse camera frame normalization
-    return (X + [1, h/w])*w/2
-    
-
-def world_to_camera(X, R, t):
-    Rt = wrap(qinverse, R) # Invert rotation
-    return wrap(qrot, np.tile(Rt, (*X.shape[:-1], 1)), X - t) # Rotate and translate
-
-    
-def camera_to_world(X, R, t):
-    return wrap(qrot, np.tile(R, (*X.shape[:-1], 1)), X) + t
-
-    
-def project_to_2d(X, camera_params):
-    """
-    Project 3D points to 2D using the Human3.6M camera projection function.
-    This is a differentiable and batched reimplementation of the original MATLAB script.
-    
-    Arguments:
-    X -- 3D points in *camera space* to transform (N, *, 3)
-    camera_params -- intrinsic parameteres (N, 2+2+3+2=9)
-    """
-    assert X.shape[-1] == 3
-    assert len(camera_params.shape) == 2
-    assert camera_params.shape[-1] == 9
-    assert X.shape[0] == camera_params.shape[0]
-    
-    while len(camera_params.shape) < len(X.shape):
-        camera_params = camera_params.unsqueeze(1)
-        
-    f = camera_params[..., :2]
-    c = camera_params[..., 2:4]
-    k = camera_params[..., 4:7]
-    p = camera_params[..., 7:]
-    
-    XX = torch.clamp(X[..., :2] / X[..., 2:], min=-1, max=1)
-    r2 = torch.sum(XX[..., :2]**2, dim=len(XX.shape)-1, keepdim=True)
-
-    radial = 1 + torch.sum(k * torch.cat((r2, r2**2, r2**3), dim=len(r2.shape)-1), dim=len(r2.shape)-1, keepdim=True)
-    tan = torch.sum(p*XX, dim=len(XX.shape)-1, keepdim=True)
-
-    XXX = XX*(radial + tan) + p*r2
-    
-    return f*XXX + c
-
-def project_to_2d_linear(X, camera_params):
-    """
-    Project 3D points to 2D using only linear parameters (focal length and principal point).
-    
-    Arguments:
-    X -- 3D points in *camera space* to transform (N, *, 3)
-    camera_params -- intrinsic parameteres (N, 2+2+3+2=9)
-    """
-    assert X.shape[-1] == 3
-    assert len(camera_params.shape) == 2
-    assert camera_params.shape[-1] == 9
-    assert X.shape[0] == camera_params.shape[0]
-    
-    while len(camera_params.shape) < len(X.shape):
-        camera_params = camera_params.unsqueeze(1)
-        
-    f = camera_params[..., :2]
-    c = camera_params[..., 2:4]
-    
-    XX = torch.clamp(X[..., :2] / X[..., 2:], min=-1, max=1)
-    
-    return f*XX + c

VideoPose3D/common/custom_dataset.py

@@ -1,66 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-import copy
-from common.skeleton import Skeleton
-from common.mocap_dataset import MocapDataset
-from common.camera import normalize_screen_coordinates, image_coordinates
-from common.h36m_dataset import h36m_skeleton
-       
-
-custom_camera_params = {
-    'id': None,
-    'res_w': None, # Pulled from metadata
-    'res_h': None, # Pulled from metadata
-    
-    # Dummy camera parameters (taken from Human3.6M), only for visualization purposes
-    'azimuth': 70, # Only used for visualization
-    'orientation': [0.1407056450843811, -0.1500701755285263, -0.755240797996521, 0.6223280429840088],
-    'translation': [1841.1070556640625, 4955.28466796875, 1563.4454345703125],
-}
-
-class CustomDataset(MocapDataset):
-    def __init__(self, detections_path, remove_static_joints=True):
-        super().__init__(fps=None, skeleton=h36m_skeleton)        
-        
-        # Load serialized dataset
-        data = np.load(detections_path, allow_pickle=True)
-        resolutions = data['metadata'].item()['video_metadata']
-        
-        self._cameras = {}
-        self._data = {}
-        for video_name, res in resolutions.items():
-            cam = {}
-            cam.update(custom_camera_params)
-            cam['orientation'] = np.array(cam['orientation'], dtype='float32')
-            cam['translation'] = np.array(cam['translation'], dtype='float32')
-            cam['translation'] = cam['translation']/1000 # mm to meters
-            
-            cam['id'] = video_name
-            cam['res_w'] = res['w']
-            cam['res_h'] = res['h']
-            
-            self._cameras[video_name] = [cam]
-        
-            self._data[video_name] = {
-                'custom': {
-                    'cameras': cam
-                }
-            }
-                
-        if remove_static_joints:
-            # Bring the skeleton to 17 joints instead of the original 32
-            self.remove_joints([4, 5, 9, 10, 11, 16, 20, 21, 22, 23, 24, 28, 29, 30, 31])
-            
-            # Rewire shoulders to the correct parents
-            self._skeleton._parents[11] = 8
-            self._skeleton._parents[14] = 8
-            
-    def supports_semi_supervised(self):
-        return False
-   

VideoPose3D/common/generators.py

@@ -1,239 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-from itertools import zip_longest
-import numpy as np
-
-class ChunkedGenerator:
-    """
-    Batched data generator, used for training.
-    The sequences are split into equal-length chunks and padded as necessary.
-    
-    Arguments:
-    batch_size -- the batch size to use for training
-    cameras -- list of cameras, one element for each video (optional, used for semi-supervised training)
-    poses_3d -- list of ground-truth 3D poses, one element for each video (optional, used for supervised training)
-    poses_2d -- list of input 2D keypoints, one element for each video
-    chunk_length -- number of output frames to predict for each training example (usually 1)
-    pad -- 2D input padding to compensate for valid convolutions, per side (depends on the receptive field)
-    causal_shift -- asymmetric padding offset when causal convolutions are used (usually 0 or "pad")
-    shuffle -- randomly shuffle the dataset before each epoch
-    random_seed -- initial seed to use for the random generator
-    augment -- augment the dataset by flipping poses horizontally
-    kps_left and kps_right -- list of left/right 2D keypoints if flipping is enabled
-    joints_left and joints_right -- list of left/right 3D joints if flipping is enabled
-    """
-    def __init__(self, batch_size, cameras, poses_3d, poses_2d,
-                 chunk_length, pad=0, causal_shift=0,
-                 shuffle=True, random_seed=1234,
-                 augment=False, kps_left=None, kps_right=None, joints_left=None, joints_right=None,
-                 endless=False):
-        assert poses_3d is None or len(poses_3d) == len(poses_2d), (len(poses_3d), len(poses_2d))
-        assert cameras is None or len(cameras) == len(poses_2d)
-    
-        # Build lineage info
-        pairs = [] # (seq_idx, start_frame, end_frame, flip) tuples
-        for i in range(len(poses_2d)):
-            assert poses_3d is None or poses_3d[i].shape[0] == poses_3d[i].shape[0]
-            n_chunks = (poses_2d[i].shape[0] + chunk_length - 1) // chunk_length
-            offset = (n_chunks * chunk_length - poses_2d[i].shape[0]) // 2
-            bounds = np.arange(n_chunks+1)*chunk_length - offset
-            augment_vector = np.full(len(bounds - 1), False, dtype=bool)
-            pairs += zip(np.repeat(i, len(bounds - 1)), bounds[:-1], bounds[1:], augment_vector)
-            if augment:
-                pairs += zip(np.repeat(i, len(bounds - 1)), bounds[:-1], bounds[1:], ~augment_vector)
-
-        # Initialize buffers
-        if cameras is not None:
-            self.batch_cam = np.empty((batch_size, cameras[0].shape[-1]))
-        if poses_3d is not None:
-            self.batch_3d = np.empty((batch_size, chunk_length, poses_3d[0].shape[-2], poses_3d[0].shape[-1]))
-        self.batch_2d = np.empty((batch_size, chunk_length + 2*pad, poses_2d[0].shape[-2], poses_2d[0].shape[-1]))
-
-        self.num_batches = (len(pairs) + batch_size - 1) // batch_size
-        self.batch_size = batch_size
-        self.random = np.random.RandomState(random_seed)
-        self.pairs = pairs
-        self.shuffle = shuffle
-        self.pad = pad
-        self.causal_shift = causal_shift
-        self.endless = endless
-        self.state = None
-        
-        self.cameras = cameras
-        self.poses_3d = poses_3d
-        self.poses_2d = poses_2d
-        
-        self.augment = augment
-        self.kps_left = kps_left
-        self.kps_right = kps_right
-        self.joints_left = joints_left
-        self.joints_right = joints_right
-        
-    def num_frames(self):
-        return self.num_batches * self.batch_size
-    
-    def random_state(self):
-        return self.random
-    
-    def set_random_state(self, random):
-        self.random = random
-        
-    def augment_enabled(self):
-        return self.augment
-    
-    def next_pairs(self):
-        if self.state is None:
-            if self.shuffle:
-                pairs = self.random.permutation(self.pairs)
-            else:
-                pairs = self.pairs
-            return 0, pairs
-        else:
-            return self.state
-    
-    def next_epoch(self):
-        enabled = True
-        while enabled:
-            start_idx, pairs = self.next_pairs()
-            for b_i in range(start_idx, self.num_batches):
-                chunks = pairs[b_i*self.batch_size : (b_i+1)*self.batch_size]
-                for i, (seq_i, start_3d, end_3d, flip) in enumerate(chunks):
-                    start_2d = start_3d - self.pad - self.causal_shift
-                    end_2d = end_3d + self.pad - self.causal_shift
-
-                    # 2D poses
-                    seq_2d = self.poses_2d[seq_i]
-                    low_2d = max(start_2d, 0)
-                    high_2d = min(end_2d, seq_2d.shape[0])
-                    pad_left_2d = low_2d - start_2d
-                    pad_right_2d = end_2d - high_2d
-                    if pad_left_2d != 0 or pad_right_2d != 0:
-                        self.batch_2d[i] = np.pad(seq_2d[low_2d:high_2d], ((pad_left_2d, pad_right_2d), (0, 0), (0, 0)), 'edge')
-                    else:
-                        self.batch_2d[i] = seq_2d[low_2d:high_2d]
-
-                    if flip:
-                        # Flip 2D keypoints
-                        self.batch_2d[i, :, :, 0] *= -1
-                        self.batch_2d[i, :, self.kps_left + self.kps_right] = self.batch_2d[i, :, self.kps_right + self.kps_left]
-
-                    # 3D poses
-                    if self.poses_3d is not None:
-                        seq_3d = self.poses_3d[seq_i]
-                        low_3d = max(start_3d, 0)
-                        high_3d = min(end_3d, seq_3d.shape[0])
-                        pad_left_3d = low_3d - start_3d
-                        pad_right_3d = end_3d - high_3d
-                        if pad_left_3d != 0 or pad_right_3d != 0:
-                            self.batch_3d[i] = np.pad(seq_3d[low_3d:high_3d], ((pad_left_3d, pad_right_3d), (0, 0), (0, 0)), 'edge')
-                        else:
-                            self.batch_3d[i] = seq_3d[low_3d:high_3d]
-
-                        if flip:
-                            # Flip 3D joints
-                            self.batch_3d[i, :, :, 0] *= -1
-                            self.batch_3d[i, :, self.joints_left + self.joints_right] = \
-                                    self.batch_3d[i, :, self.joints_right + self.joints_left]
-
-                    # Cameras
-                    if self.cameras is not None:
-                        self.batch_cam[i] = self.cameras[seq_i]
-                        if flip:
-                            # Flip horizontal distortion coefficients
-                            self.batch_cam[i, 2] *= -1
-                            self.batch_cam[i, 7] *= -1
-
-                if self.endless:
-                    self.state = (b_i + 1, pairs)
-                if self.poses_3d is None and self.cameras is None:
-                    yield None, None, self.batch_2d[:len(chunks)]
-                elif self.poses_3d is not None and self.cameras is None:
-                    yield None, self.batch_3d[:len(chunks)], self.batch_2d[:len(chunks)]
-                elif self.poses_3d is None:
-                    yield self.batch_cam[:len(chunks)], None, self.batch_2d[:len(chunks)]
-                else:
-                    yield self.batch_cam[:len(chunks)], self.batch_3d[:len(chunks)], self.batch_2d[:len(chunks)]
-            
-            if self.endless:
-                self.state = None
-            else:
-                enabled = False
-            
-
-class UnchunkedGenerator:
-    """
-    Non-batched data generator, used for testing.
-    Sequences are returned one at a time (i.e. batch size = 1), without chunking.
-    
-    If data augmentation is enabled, the batches contain two sequences (i.e. batch size = 2),
-    the second of which is a mirrored version of the first.
-    
-    Arguments:
-    cameras -- list of cameras, one element for each video (optional, used for semi-supervised training)
-    poses_3d -- list of ground-truth 3D poses, one element for each video (optional, used for supervised training)
-    poses_2d -- list of input 2D keypoints, one element for each video
-    pad -- 2D input padding to compensate for valid convolutions, per side (depends on the receptive field)
-    causal_shift -- asymmetric padding offset when causal convolutions are used (usually 0 or "pad")
-    augment -- augment the dataset by flipping poses horizontally
-    kps_left and kps_right -- list of left/right 2D keypoints if flipping is enabled
-    joints_left and joints_right -- list of left/right 3D joints if flipping is enabled
-    """
-    
-    def __init__(self, cameras, poses_3d, poses_2d, pad=0, causal_shift=0,
-                 augment=False, kps_left=None, kps_right=None, joints_left=None, joints_right=None):
-        assert poses_3d is None or len(poses_3d) == len(poses_2d)
-        assert cameras is None or len(cameras) == len(poses_2d)
-
-        self.augment = augment
-        self.kps_left = kps_left
-        self.kps_right = kps_right
-        self.joints_left = joints_left
-        self.joints_right = joints_right
-        
-        self.pad = pad
-        self.causal_shift = causal_shift
-        self.cameras = [] if cameras is None else cameras
-        self.poses_3d = [] if poses_3d is None else poses_3d
-        self.poses_2d = poses_2d
-        
-    def num_frames(self):
-        count = 0
-        for p in self.poses_2d:
-            count += p.shape[0]
-        return count
-    
-    def augment_enabled(self):
-        return self.augment
-    
-    def set_augment(self, augment):
-        self.augment = augment
-    
-    def next_epoch(self):
-        for seq_cam, seq_3d, seq_2d in zip_longest(self.cameras, self.poses_3d, self.poses_2d):
-            batch_cam = None if seq_cam is None else np.expand_dims(seq_cam, axis=0)
-            batch_3d = None if seq_3d is None else np.expand_dims(seq_3d, axis=0)
-            batch_2d = np.expand_dims(np.pad(seq_2d,
-                            ((self.pad + self.causal_shift, self.pad - self.causal_shift), (0, 0), (0, 0)),
-                            'edge'), axis=0)
-            if self.augment:
-                # Append flipped version
-                if batch_cam is not None:
-                    batch_cam = np.concatenate((batch_cam, batch_cam), axis=0)
-                    batch_cam[1, 2] *= -1
-                    batch_cam[1, 7] *= -1
-                
-                if batch_3d is not None:
-                    batch_3d = np.concatenate((batch_3d, batch_3d), axis=0)
-                    batch_3d[1, :, :, 0] *= -1
-                    batch_3d[1, :, self.joints_left + self.joints_right] = batch_3d[1, :, self.joints_right + self.joints_left]
-
-                batch_2d = np.concatenate((batch_2d, batch_2d), axis=0)
-                batch_2d[1, :, :, 0] *= -1
-                batch_2d[1, :, self.kps_left + self.kps_right] = batch_2d[1, :, self.kps_right + self.kps_left]
-
-            yield batch_cam, batch_3d, batch_2d

VideoPose3D/common/h36m_dataset.py

@@ -1,255 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-import copy
-from common.skeleton import Skeleton
-from common.mocap_dataset import MocapDataset
-from common.camera import normalize_screen_coordinates, image_coordinates
-       
-h36m_skeleton = Skeleton(parents=[-1,  0,  1,  2,  3,  4,  0,  6,  7,  8,  9,  0, 11, 12, 13, 14, 12,
-       16, 17, 18, 19, 20, 19, 22, 12, 24, 25, 26, 27, 28, 27, 30],
-       joints_left=[6, 7, 8, 9, 10, 16, 17, 18, 19, 20, 21, 22, 23],
-       joints_right=[1, 2, 3, 4, 5, 24, 25, 26, 27, 28, 29, 30, 31])
-
-h36m_cameras_intrinsic_params = [
-    {
-        'id': '54138969',
-        'center': [512.54150390625, 515.4514770507812],
-        'focal_length': [1145.0494384765625, 1143.7811279296875],
-        'radial_distortion': [-0.20709891617298126, 0.24777518212795258, -0.0030751503072679043],
-        'tangential_distortion': [-0.0009756988729350269, -0.00142447161488235],
-        'res_w': 1000,
-        'res_h': 1002,
-        'azimuth': 70, # Only used for visualization
-    },
-    {
-        'id': '55011271',
-        'center': [508.8486328125, 508.0649108886719],
-        'focal_length': [1149.6756591796875, 1147.5916748046875],
-        'radial_distortion': [-0.1942136287689209, 0.2404085397720337, 0.006819975562393665],
-        'tangential_distortion': [-0.0016190266469493508, -0.0027408944442868233],
-        'res_w': 1000,
-        'res_h': 1000,
-        'azimuth': -70, # Only used for visualization
-    },
-    {
-        'id': '58860488',
-        'center': [519.8158569335938, 501.40264892578125],
-        'focal_length': [1149.1407470703125, 1148.7989501953125],
-        'radial_distortion': [-0.2083381861448288, 0.25548800826072693, -0.0024604974314570427],
-        'tangential_distortion': [0.0014843869721516967, -0.0007599993259645998],
-        'res_w': 1000,
-        'res_h': 1000,
-        'azimuth': 110, # Only used for visualization
-    },
-    {
-        'id': '60457274',
-        'center': [514.9682006835938, 501.88201904296875],
-        'focal_length': [1145.5113525390625, 1144.77392578125],
-        'radial_distortion': [-0.198384091258049, 0.21832367777824402, -0.008947807364165783],
-        'tangential_distortion': [-0.0005872055771760643, -0.0018133620033040643],
-        'res_w': 1000,
-        'res_h': 1002,
-        'azimuth': -110, # Only used for visualization
-    },
-]
-
-h36m_cameras_extrinsic_params = {
-    'S1': [
-        {
-            'orientation': [0.1407056450843811, -0.1500701755285263, -0.755240797996521, 0.6223280429840088],
-            'translation': [1841.1070556640625, 4955.28466796875, 1563.4454345703125],
-        },
-        {
-            'orientation': [0.6157187819480896, -0.764836311340332, -0.14833825826644897, 0.11794740706682205],
-            'translation': [1761.278564453125, -5078.0068359375, 1606.2650146484375],
-        },
-        {
-            'orientation': [0.14651472866535187, -0.14647851884365082, 0.7653023600578308, -0.6094175577163696],
-            'translation': [-1846.7777099609375, 5215.04638671875, 1491.972412109375],
-        },
-        {
-            'orientation': [0.5834008455276489, -0.7853162288665771, 0.14548823237419128, -0.14749594032764435],
-            'translation': [-1794.7896728515625, -3722.698974609375, 1574.8927001953125],
-        },
-    ],
-    'S2': [
-        {},
-        {},
-        {},
-        {},
-    ],
-    'S3': [
-        {},
-        {},
-        {},
-        {},
-    ],
-    'S4': [
-        {},
-        {},
-        {},
-        {},
-    ],
-    'S5': [
-        {
-            'orientation': [0.1467377245426178, -0.162370964884758, -0.7551892995834351, 0.6178938746452332],
-            'translation': [2097.3916015625, 4880.94482421875, 1605.732421875],
-        },
-        {
-            'orientation': [0.6159758567810059, -0.7626792192459106, -0.15728192031383514, 0.1189815029501915],
-            'translation': [2031.7008056640625, -5167.93310546875, 1612.923095703125],
-        },
-        {
-            'orientation': [0.14291371405124664, -0.12907841801643372, 0.7678384780883789, -0.6110143065452576],
-            'translation': [-1620.5948486328125, 5171.65869140625, 1496.43701171875],
-        },
-        {
-            'orientation': [0.5920479893684387, -0.7814217805862427, 0.1274748593568802, -0.15036417543888092],
-            'translation': [-1637.1737060546875, -3867.3173828125, 1547.033203125],
-        },
-    ],
-    'S6': [
-        {
-            'orientation': [0.1337897777557373, -0.15692396461963654, -0.7571090459823608, 0.6198879480361938],
-            'translation': [1935.4517822265625, 4950.24560546875, 1618.0838623046875],
-        },
-        {
-            'orientation': [0.6147197484970093, -0.7628812789916992, -0.16174767911434174, 0.11819244921207428],
-            'translation': [1969.803955078125, -5128.73876953125, 1632.77880859375],
-        },
-        {
-            'orientation': [0.1529948115348816, -0.13529130816459656, 0.7646096348762512, -0.6112781167030334],
-            'translation': [-1769.596435546875, 5185.361328125, 1476.993408203125],
-        },
-        {
-            'orientation': [0.5916101336479187, -0.7804774045944214, 0.12832270562648773, -0.1561593860387802],
-            'translation': [-1721.668701171875, -3884.13134765625, 1540.4879150390625],
-        },
-    ],
-    'S7': [
-        {
-            'orientation': [0.1435241848230362, -0.1631336808204651, -0.7548328638076782, 0.6188824772834778],
-            'translation': [1974.512939453125, 4926.3544921875, 1597.8326416015625],
-        },
-        {
-            'orientation': [0.6141672730445862, -0.7638262510299683, -0.1596645563840866, 0.1177929937839508],
-            'translation': [1937.0584716796875, -5119.7900390625, 1631.5665283203125],
-        },
-        {
-            'orientation': [0.14550060033798218, -0.12874816358089447, 0.7660516500473022, -0.6127139329910278],
-            'translation': [-1741.8111572265625, 5208.24951171875, 1464.8245849609375],
-        },
-        {
-            'orientation': [0.5912848114967346, -0.7821764349937439, 0.12445473670959473, -0.15196487307548523],
-            'translation': [-1734.7105712890625, -3832.42138671875, 1548.5830078125],
-        },
-    ],
-    'S8': [
-        {
-            'orientation': [0.14110587537288666, -0.15589867532253265, -0.7561917304992676, 0.619644045829773],
-            'translation': [2150.65185546875, 4896.1611328125, 1611.9046630859375],
-        },
-        {
-            'orientation': [0.6169601678848267, -0.7647668123245239, -0.14846350252628326, 0.11158157885074615],
-            'translation': [2219.965576171875, -5148.453125, 1613.0440673828125],
-        },
-        {
-            'orientation': [0.1471444070339203, -0.13377119600772858, 0.7670128345489502, -0.6100369691848755],
-            'translation': [-1571.2215576171875, 5137.0185546875, 1498.1761474609375],
-        },
-        {
-            'orientation': [0.5927824378013611, -0.7825870513916016, 0.12147816270589828, -0.14631995558738708],
-            'translation': [-1476.913330078125, -3896.7412109375, 1547.97216796875],
-        },
-    ],
-    'S9': [
-        {
-            'orientation': [0.15540587902069092, -0.15548215806484222, -0.7532095313072205, 0.6199594736099243],
-            'translation': [2044.45849609375, 4935.1171875, 1481.2275390625],
-        },
-        {
-            'orientation': [0.618784487247467, -0.7634735107421875, -0.14132238924503326, 0.11933968216180801],
-            'translation': [1990.959716796875, -5123.810546875, 1568.8048095703125],
-        },
-        {
-            'orientation': [0.13357827067375183, -0.1367100477218628, 0.7689454555511475, -0.6100738644599915],
-            'translation': [-1670.9921875, 5211.98583984375, 1528.387939453125],
-        },
-        {
-            'orientation': [0.5879399180412292, -0.7823407053947449, 0.1427614390850067, -0.14794869720935822],
-            'translation': [-1696.04345703125, -3827.099853515625, 1591.4127197265625],
-        },
-    ],
-    'S11': [
-        {
-            'orientation': [0.15232472121715546, -0.15442320704460144, -0.7547563314437866, 0.6191070079803467],
-            'translation': [2098.440185546875, 4926.5546875, 1500.278564453125],
-        },
-        {
-            'orientation': [0.6189449429512024, -0.7600917220115662, -0.15300633013248444, 0.1255258321762085],
-            'translation': [2083.182373046875, -4912.1728515625, 1561.07861328125],
-        },
-        {
-            'orientation': [0.14943228662014008, -0.15650227665901184, 0.7681233882904053, -0.6026304364204407],
-            'translation': [-1609.8153076171875, 5177.3359375, 1537.896728515625],
-        },
-        {
-            'orientation': [0.5894251465797424, -0.7818877100944519, 0.13991211354732513, -0.14715361595153809],
-            'translation': [-1590.738037109375, -3854.1689453125, 1578.017578125],
-        },
-    ],
-}
-
-class Human36mDataset(MocapDataset):
-    def __init__(self, path, remove_static_joints=True):
-        super().__init__(fps=50, skeleton=h36m_skeleton)
-        
-        self._cameras = copy.deepcopy(h36m_cameras_extrinsic_params)
-        for cameras in self._cameras.values():
-            for i, cam in enumerate(cameras):
-                cam.update(h36m_cameras_intrinsic_params[i])
-                for k, v in cam.items():
-                    if k not in ['id', 'res_w', 'res_h']:
-                        cam[k] = np.array(v, dtype='float32')
-                
-                # Normalize camera frame
-                cam['center'] = normalize_screen_coordinates(cam['center'], w=cam['res_w'], h=cam['res_h']).astype('float32')
-                cam['focal_length'] = cam['focal_length']/cam['res_w']*2
-                if 'translation' in cam:
-                    cam['translation'] = cam['translation']/1000 # mm to meters
-                
-                # Add intrinsic parameters vector
-                cam['intrinsic'] = np.concatenate((cam['focal_length'],
-                                                   cam['center'],
-                                                   cam['radial_distortion'],
-                                                   cam['tangential_distortion']))
-        
-        # Load serialized dataset
-        data = np.load(path, allow_pickle=True)['positions_3d'].item()
-        
-        self._data = {}
-        for subject, actions in data.items():
-            self._data[subject] = {}
-            for action_name, positions in actions.items():
-                self._data[subject][action_name] = {
-                    'positions': positions,
-                    'cameras': self._cameras[subject],
-                }
-                
-        if remove_static_joints:
-            # Bring the skeleton to 17 joints instead of the original 32
-            self.remove_joints([4, 5, 9, 10, 11, 16, 20, 21, 22, 23, 24, 28, 29, 30, 31])
-            
-            # Rewire shoulders to the correct parents
-            self._skeleton._parents[11] = 8
-            self._skeleton._parents[14] = 8
-            
-    def supports_semi_supervised(self):
-        return True
-   

VideoPose3D/common/humaneva_dataset.py

@@ -1,121 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-import copy
-from common.skeleton import Skeleton
-from common.mocap_dataset import MocapDataset
-from common.camera import normalize_screen_coordinates, image_coordinates
-       
-humaneva_skeleton = Skeleton(parents=[-1, 0, 1, 2, 3, 1, 5, 6, 0, 8, 9, 0, 11, 12, 1],
-       joints_left=[2, 3, 4, 8, 9, 10],
-       joints_right=[5, 6, 7, 11, 12, 13])
-
-humaneva_cameras_intrinsic_params = [
-    {
-        'id': 'C1',
-        'res_w': 640,
-        'res_h': 480,
-        'azimuth': 0, # Only used for visualization
-    },
-    {
-        'id': 'C2',
-        'res_w': 640,
-        'res_h': 480,
-        'azimuth': -90, # Only used for visualization
-    },
-    {
-        'id': 'C3',
-        'res_w': 640,
-        'res_h': 480,
-        'azimuth': 90, # Only used for visualization
-    },
-]
-
-humaneva_cameras_extrinsic_params = {
-    'S1': [
-        {
-            'orientation': [0.424207, -0.4983646, -0.5802981, 0.4847012],
-            'translation': [4062.227,  663.2477, 1528.397],
-        },
-        {
-            'orientation': [0.6503354, -0.7481602, -0.0919284, 0.0941766],
-            'translation': [844.8131, -3805.2092,  1504.9929],
-        },
-        {
-            'orientation': [0.0664734, -0.0690535, 0.7416416, -0.6639132],
-            'translation': [-797.67377, 3916.3174, 1433.6602],
-        },
-    ],
-    'S2': [
-        {
-            'orientation': [ 0.4214752, -0.4961493, -0.5838273, 0.4851187 ],
-            'translation': [ 4112.9121,   626.4929,  1545.2988], 
-        },
-        {
-            'orientation': [ 0.6501393, -0.7476588, -0.0954617, 0.0959808 ],
-            'translation': [  923.5740, -3877.9243,  1504.5518], 
-        },
-        {
-            'orientation': [ 0.0699353, -0.0712403, 0.7421637, -0.662742 ],
-            'translation': [ -781.4915,  3838.8853,  1444.9929], 
-        },
-    ],
-    'S3': [
-        {
-            'orientation': [ 0.424207, -0.4983646, -0.5802981, 0.4847012 ],
-            'translation': [ 4062.2271,   663.2477,  1528.3970], 
-        },
-        {
-            'orientation': [ 0.6503354, -0.7481602, -0.0919284, 0.0941766 ],
-            'translation': [  844.8131, -3805.2092,  1504.9929], 
-        },
-        {
-            'orientation': [ 0.0664734, -0.0690535, 0.7416416, -0.6639132 ],
-            'translation': [ -797.6738,  3916.3174,  1433.6602], 
-        },
-    ],
-    'S4': [
-        {},
-        {},
-        {},
-    ],
-    
-}
-
-class HumanEvaDataset(MocapDataset):
-    def __init__(self, path):
-        super().__init__(fps=60, skeleton=humaneva_skeleton)
-        
-        self._cameras = copy.deepcopy(humaneva_cameras_extrinsic_params)
-        for cameras in self._cameras.values():
-            for i, cam in enumerate(cameras):
-                cam.update(humaneva_cameras_intrinsic_params[i])
-                for k, v in cam.items():
-                    if k not in ['id', 'res_w', 'res_h']:
-                        cam[k] = np.array(v, dtype='float32')
-                if 'translation' in cam:
-                    cam['translation'] = cam['translation']/1000 # mm to meters
-                
-        for subject in list(self._cameras.keys()):
-            data = self._cameras[subject]
-            del self._cameras[subject]
-            for prefix in ['Train/', 'Validate/', 'Unlabeled/Train/', 'Unlabeled/Validate/', 'Unlabeled/']:
-                self._cameras[prefix + subject] = data
-        
-        # Load serialized dataset
-        data = np.load(path, allow_pickle=True)['positions_3d'].item()
-        
-        self._data = {}
-        for subject, actions in data.items():
-            self._data[subject] = {}
-            for action_name, positions in actions.items():
-                self._data[subject][action_name] = {
-                    'positions': positions,
-                    'cameras': self._cameras[subject],
-                }
-   

VideoPose3D/common/loss.py

@@ -1,89 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import torch
-import numpy as np
-
-def mpjpe(predicted, target):
-    """
-    Mean per-joint position error (i.e. mean Euclidean distance),
-    often referred to as "Protocol #1" in many papers.
-    """
-    assert predicted.shape == target.shape
-    return torch.mean(torch.norm(predicted - target, dim=len(target.shape)-1))
-    
-def weighted_mpjpe(predicted, target, w):
-    """
-    Weighted mean per-joint position error (i.e. mean Euclidean distance)
-    """
-    assert predicted.shape == target.shape
-    assert w.shape[0] == predicted.shape[0]
-    return torch.mean(w * torch.norm(predicted - target, dim=len(target.shape)-1))
-
-def p_mpjpe(predicted, target):
-    """
-    Pose error: MPJPE after rigid alignment (scale, rotation, and translation),
-    often referred to as "Protocol #2" in many papers.
-    """
-    assert predicted.shape == target.shape
-    
-    muX = np.mean(target, axis=1, keepdims=True)
-    muY = np.mean(predicted, axis=1, keepdims=True)
-    
-    X0 = target - muX
-    Y0 = predicted - muY
-
-    normX = np.sqrt(np.sum(X0**2, axis=(1, 2), keepdims=True))
-    normY = np.sqrt(np.sum(Y0**2, axis=(1, 2), keepdims=True))
-    
-    X0 /= normX
-    Y0 /= normY
-
-    H = np.matmul(X0.transpose(0, 2, 1), Y0)
-    U, s, Vt = np.linalg.svd(H)
-    V = Vt.transpose(0, 2, 1)
-    R = np.matmul(V, U.transpose(0, 2, 1))
-
-    # Avoid improper rotations (reflections), i.e. rotations with det(R) = -1
-    sign_detR = np.sign(np.expand_dims(np.linalg.det(R), axis=1))
-    V[:, :, -1] *= sign_detR
-    s[:, -1] *= sign_detR.flatten()
-    R = np.matmul(V, U.transpose(0, 2, 1)) # Rotation
-
-    tr = np.expand_dims(np.sum(s, axis=1, keepdims=True), axis=2)
-
-    a = tr * normX / normY # Scale
-    t = muX - a*np.matmul(muY, R) # Translation
-    
-    # Perform rigid transformation on the input
-    predicted_aligned = a*np.matmul(predicted, R) + t
-    
-    # Return MPJPE
-    return np.mean(np.linalg.norm(predicted_aligned - target, axis=len(target.shape)-1))
-    
-def n_mpjpe(predicted, target):
-    """
-    Normalized MPJPE (scale only), adapted from:
-    https://github.com/hrhodin/UnsupervisedGeometryAwareRepresentationLearning/blob/master/losses/poses.py
-    """
-    assert predicted.shape == target.shape
-    
-    norm_predicted = torch.mean(torch.sum(predicted**2, dim=3, keepdim=True), dim=2, keepdim=True)
-    norm_target = torch.mean(torch.sum(target*predicted, dim=3, keepdim=True), dim=2, keepdim=True)
-    scale = norm_target / norm_predicted
-    return mpjpe(scale * predicted, target)
-
-def mean_velocity_error(predicted, target):
-    """
-    Mean per-joint velocity error (i.e. mean Euclidean distance of the 1st derivative)
-    """
-    assert predicted.shape == target.shape
-    
-    velocity_predicted = np.diff(predicted, axis=0)
-    velocity_target = np.diff(target, axis=0)
-    
-    return np.mean(np.linalg.norm(velocity_predicted - velocity_target, axis=len(target.shape)-1))

VideoPose3D/common/mocap_dataset.py

@@ -1,44 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-from common.skeleton import Skeleton
-
-class MocapDataset:
-    def __init__(self, fps, skeleton):
-        self._skeleton = skeleton
-        self._fps = fps
-        self._data = None # Must be filled by subclass
-        self._cameras = None # Must be filled by subclass
-    
-    def remove_joints(self, joints_to_remove):
-        kept_joints = self._skeleton.remove_joints(joints_to_remove)
-        for subject in self._data.keys():
-            for action in self._data[subject].keys():
-                s = self._data[subject][action]
-                if 'positions' in s:
-                    s['positions'] = s['positions'][:, kept_joints]
-                
-        
-    def __getitem__(self, key):
-        return self._data[key]
-        
-    def subjects(self):
-        return self._data.keys()
-    
-    def fps(self):
-        return self._fps
-    
-    def skeleton(self):
-        return self._skeleton
-        
-    def cameras(self):
-        return self._cameras
-    
-    def supports_semi_supervised(self):
-        # This method can be overridden
-        return False

VideoPose3D/common/model.py

@@ -1,197 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import torch.nn as nn
-
-class TemporalModelBase(nn.Module):
-    """
-    Do not instantiate this class.
-    """
-    
-    def __init__(self, num_joints_in, in_features, num_joints_out,
-                 filter_widths, causal, dropout, channels):
-        super().__init__()
-        
-        # Validate input
-        for fw in filter_widths:
-            assert fw % 2 != 0, 'Only odd filter widths are supported'
-        
-        self.num_joints_in = num_joints_in
-        self.in_features = in_features
-        self.num_joints_out = num_joints_out
-        self.filter_widths = filter_widths
-        
-        self.drop = nn.Dropout(dropout)
-        self.relu = nn.ReLU(inplace=True)
-        
-        self.pad = [ filter_widths[0] // 2 ]
-        self.expand_bn = nn.BatchNorm1d(channels, momentum=0.1)
-        self.shrink = nn.Conv1d(channels, num_joints_out*3, 1)
-        
-
-    def set_bn_momentum(self, momentum):
-        self.expand_bn.momentum = momentum
-        for bn in self.layers_bn:
-            bn.momentum = momentum
-            
-    def receptive_field(self):
-        """
-        Return the total receptive field of this model as # of frames.
-        """
-        frames = 0
-        for f in self.pad:
-            frames += f
-        return 1 + 2*frames
-    
-    def total_causal_shift(self):
-        """
-        Return the asymmetric offset for sequence padding.
-        The returned value is typically 0 if causal convolutions are disabled,
-        otherwise it is half the receptive field.
-        """
-        frames = self.causal_shift[0]
-        next_dilation = self.filter_widths[0]
-        for i in range(1, len(self.filter_widths)):
-            frames += self.causal_shift[i] * next_dilation
-            next_dilation *= self.filter_widths[i]
-        return frames
-        
-    def forward(self, x):
-        assert len(x.shape) == 4
-        assert x.shape[-2] == self.num_joints_in
-        assert x.shape[-1] == self.in_features
-        
-        sz = x.shape[:3]
-        x = x.view(x.shape[0], x.shape[1], -1)
-        x = x.permute(0, 2, 1)
-        
-        x = self._forward_blocks(x)
-        
-        x = x.permute(0, 2, 1)
-        x = x.view(sz[0], -1, self.num_joints_out, 3)
-        
-        return x    
-
-class TemporalModel(TemporalModelBase):
-    """
-    Reference 3D pose estimation model with temporal convolutions.
-    This implementation can be used for all use-cases.
-    """
-    
-    def __init__(self, num_joints_in, in_features, num_joints_out,
-                 filter_widths, causal=False, dropout=0.25, channels=1024, dense=False):
-        """
-        Initialize this model.
-        
-        Arguments:
-        num_joints_in -- number of input joints (e.g. 17 for Human3.6M)
-        in_features -- number of input features for each joint (typically 2 for 2D input)
-        num_joints_out -- number of output joints (can be different than input)
-        filter_widths -- list of convolution widths, which also determines the # of blocks and receptive field
-        causal -- use causal convolutions instead of symmetric convolutions (for real-time applications)
-        dropout -- dropout probability
-        channels -- number of convolution channels
-        dense -- use regular dense convolutions instead of dilated convolutions (ablation experiment)
-        """
-        super().__init__(num_joints_in, in_features, num_joints_out, filter_widths, causal, dropout, channels)
-        
-        self.expand_conv = nn.Conv1d(num_joints_in*in_features, channels, filter_widths[0], bias=False)
-        
-        layers_conv = []
-        layers_bn = []
-        
-        self.causal_shift = [ (filter_widths[0]) // 2 if causal else 0 ]
-        next_dilation = filter_widths[0]
-        for i in range(1, len(filter_widths)):
-            self.pad.append((filter_widths[i] - 1)*next_dilation // 2)
-            self.causal_shift.append((filter_widths[i]//2 * next_dilation) if causal else 0)
-            
-            layers_conv.append(nn.Conv1d(channels, channels,
-                                         filter_widths[i] if not dense else (2*self.pad[-1] + 1),
-                                         dilation=next_dilation if not dense else 1,
-                                         bias=False))
-            layers_bn.append(nn.BatchNorm1d(channels, momentum=0.1))
-            layers_conv.append(nn.Conv1d(channels, channels, 1, dilation=1, bias=False))
-            layers_bn.append(nn.BatchNorm1d(channels, momentum=0.1))
-            
-            next_dilation *= filter_widths[i]
-            
-        self.layers_conv = nn.ModuleList(layers_conv)
-        self.layers_bn = nn.ModuleList(layers_bn)
-        
-    def _forward_blocks(self, x):
-        x = self.drop(self.relu(self.expand_bn(self.expand_conv(x))))
-        
-        for i in range(len(self.pad) - 1):
-            pad = self.pad[i+1]
-            shift = self.causal_shift[i+1]
-            res = x[:, :, pad + shift : x.shape[2] - pad + shift]
-            
-            x = self.drop(self.relu(self.layers_bn[2*i](self.layers_conv[2*i](x))))
-            x = res + self.drop(self.relu(self.layers_bn[2*i + 1](self.layers_conv[2*i + 1](x))))
-        
-        x = self.shrink(x)
-        return x
-    
-class TemporalModelOptimized1f(TemporalModelBase):
-    """
-    3D pose estimation model optimized for single-frame batching, i.e.
-    where batches have input length = receptive field, and output length = 1.
-    This scenario is only used for training when stride == 1.
-    
-    This implementation replaces dilated convolutions with strided convolutions
-    to avoid generating unused intermediate results. The weights are interchangeable
-    with the reference implementation.
-    """
-    
-    def __init__(self, num_joints_in, in_features, num_joints_out,
-                 filter_widths, causal=False, dropout=0.25, channels=1024):
-        """
-        Initialize this model.
-        
-        Arguments:
-        num_joints_in -- number of input joints (e.g. 17 for Human3.6M)
-        in_features -- number of input features for each joint (typically 2 for 2D input)
-        num_joints_out -- number of output joints (can be different than input)
-        filter_widths -- list of convolution widths, which also determines the # of blocks and receptive field
-        causal -- use causal convolutions instead of symmetric convolutions (for real-time applications)
-        dropout -- dropout probability
-        channels -- number of convolution channels
-        """
-        super().__init__(num_joints_in, in_features, num_joints_out, filter_widths, causal, dropout, channels)
-        
-        self.expand_conv = nn.Conv1d(num_joints_in*in_features, channels, filter_widths[0], stride=filter_widths[0], bias=False)
-        
-        layers_conv = []
-        layers_bn = []
-        
-        self.causal_shift = [ (filter_widths[0] // 2) if causal else 0 ]
-        next_dilation = filter_widths[0]
-        for i in range(1, len(filter_widths)):
-            self.pad.append((filter_widths[i] - 1)*next_dilation // 2)
-            self.causal_shift.append((filter_widths[i]//2) if causal else 0)
-            
-            layers_conv.append(nn.Conv1d(channels, channels, filter_widths[i], stride=filter_widths[i], bias=False))
-            layers_bn.append(nn.BatchNorm1d(channels, momentum=0.1))
-            layers_conv.append(nn.Conv1d(channels, channels, 1, dilation=1, bias=False))
-            layers_bn.append(nn.BatchNorm1d(channels, momentum=0.1))
-            next_dilation *= filter_widths[i]
-            
-        self.layers_conv = nn.ModuleList(layers_conv)
-        self.layers_bn = nn.ModuleList(layers_bn)
-        
-    def _forward_blocks(self, x):
-        x = self.drop(self.relu(self.expand_bn(self.expand_conv(x))))
-        
-        for i in range(len(self.pad) - 1):
-            res = x[:, :, self.causal_shift[i+1] + self.filter_widths[i+1]//2 :: self.filter_widths[i+1]]
-            
-            x = self.drop(self.relu(self.layers_bn[2*i](self.layers_conv[2*i](x))))
-            x = res + self.drop(self.relu(self.layers_bn[2*i + 1](self.layers_conv[2*i + 1](x))))
-        
-        x = self.shrink(x)
-        return x

VideoPose3D/common/quaternion.py

@@ -1,35 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import torch
-
-def qrot(q, v):
-    """
-    Rotate vector(s) v about the rotation described by quaternion(s) q.
-    Expects a tensor of shape (*, 4) for q and a tensor of shape (*, 3) for v,
-    where * denotes any number of dimensions.
-    Returns a tensor of shape (*, 3).
-    """
-    assert q.shape[-1] == 4
-    assert v.shape[-1] == 3
-    assert q.shape[:-1] == v.shape[:-1]
-
-    qvec = q[..., 1:]
-    uv = torch.cross(qvec, v, dim=len(q.shape)-1)
-    uuv = torch.cross(qvec, uv, dim=len(q.shape)-1)
-    return (v + 2 * (q[..., :1] * uv + uuv))
-    
-    
-def qinverse(q, inplace=False):
-    # We assume the quaternion to be normalized
-    if inplace:
-        q[..., 1:] *= -1
-        return q
-    else:
-        w = q[..., :1]
-        xyz = q[..., 1:]
-        return torch.cat((w, -xyz), dim=len(q.shape)-1)

VideoPose3D/common/skeleton.py

@@ -1,88 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-
-class Skeleton:
-    def __init__(self, parents, joints_left, joints_right):
-        assert len(joints_left) == len(joints_right)
-        
-        self._parents = np.array(parents)
-        self._joints_left = joints_left
-        self._joints_right = joints_right
-        self._compute_metadata()
-    
-    def num_joints(self):
-        return len(self._parents)
-    
-    def parents(self):
-        return self._parents
-    
-    def has_children(self):
-        return self._has_children
-    
-    def children(self):
-        return self._children
-    
-    def remove_joints(self, joints_to_remove):
-        """
-        Remove the joints specified in 'joints_to_remove'.
-        """
-        valid_joints = []
-        for joint in range(len(self._parents)):
-            if joint not in joints_to_remove:
-                valid_joints.append(joint)
-
-        for i in range(len(self._parents)):
-            while self._parents[i] in joints_to_remove:
-                self._parents[i] = self._parents[self._parents[i]]
-                
-        index_offsets = np.zeros(len(self._parents), dtype=int)
-        new_parents = []
-        for i, parent in enumerate(self._parents):
-            if i not in joints_to_remove:
-                new_parents.append(parent - index_offsets[parent])
-            else:
-                index_offsets[i:] += 1
-        self._parents = np.array(new_parents)
-        
-        
-        if self._joints_left is not None:
-            new_joints_left = []
-            for joint in self._joints_left:
-                if joint in valid_joints:
-                    new_joints_left.append(joint - index_offsets[joint])
-            self._joints_left = new_joints_left
-        if self._joints_right is not None:
-            new_joints_right = []
-            for joint in self._joints_right:
-                if joint in valid_joints:
-                    new_joints_right.append(joint - index_offsets[joint])
-            self._joints_right = new_joints_right
-
-        self._compute_metadata()
-        
-        return valid_joints
-    
-    def joints_left(self):
-        return self._joints_left
-    
-    def joints_right(self):
-        return self._joints_right
-        
-    def _compute_metadata(self):
-        self._has_children = np.zeros(len(self._parents)).astype(bool)
-        for i, parent in enumerate(self._parents):
-            if parent != -1:
-                self._has_children[parent] = True
-
-        self._children = []
-        for i, parent in enumerate(self._parents):
-            self._children.append([])
-        for i, parent in enumerate(self._parents):
-            if parent != -1:
-                self._children[parent].append(i)

VideoPose3D/common/utils.py

@@ -1,47 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import torch
-import numpy as np
-import hashlib
-
-def wrap(func, *args, unsqueeze=False):
-    """
-    Wrap a torch function so it can be called with NumPy arrays.
-    Input and return types are seamlessly converted.
-    """
-    
-    # Convert input types where applicable
-    args = list(args)
-    for i, arg in enumerate(args):
-        if type(arg) == np.ndarray:
-            args[i] = torch.from_numpy(arg)
-            if unsqueeze:
-                args[i] = args[i].unsqueeze(0)
-        
-    result = func(*args)
-    
-    # Convert output types where applicable
-    if isinstance(result, tuple):
-        result = list(result)
-        for i, res in enumerate(result):
-            if type(res) == torch.Tensor:
-                if unsqueeze:
-                    res = res.squeeze(0)
-                result[i] = res.numpy()
-        return tuple(result)
-    elif type(result) == torch.Tensor:
-        if unsqueeze:
-            result = result.squeeze(0)
-        return result.numpy()
-    else:
-        return result
-    
-def deterministic_random(min_value, max_value, data):
-    digest = hashlib.sha256(data.encode()).digest()
-    raw_value = int.from_bytes(digest[:4], byteorder='little', signed=False)
-    return int(raw_value / (2**32 - 1) * (max_value - min_value)) + min_value

VideoPose3D/common/visualization.py

@@ -1,209 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import matplotlib
-matplotlib.use('Agg')
-
-import matplotlib.pyplot as plt
-from matplotlib.animation import FuncAnimation, writers
-from mpl_toolkits.mplot3d import Axes3D
-import numpy as np
-import subprocess as sp
-
-def get_resolution(filename):
-    command = ['ffprobe', '-v', 'error', '-select_streams', 'v:0',
-               '-show_entries', 'stream=width,height', '-of', 'csv=p=0', filename]
-    with sp.Popen(command, stdout=sp.PIPE, bufsize=-1) as pipe:
-        for line in pipe.stdout:
-            w, h = line.decode().strip().split(',')
-            return int(w), int(h)
-            
-def get_fps(filename):
-    command = ['ffprobe', '-v', 'error', '-select_streams', 'v:0',
-               '-show_entries', 'stream=r_frame_rate', '-of', 'csv=p=0', filename]
-    with sp.Popen(command, stdout=sp.PIPE, bufsize=-1) as pipe:
-        for line in pipe.stdout:
-            a, b = line.decode().strip().split('/')
-            return int(a) / int(b)
-
-def read_video(filename, skip=0, limit=-1):
-    w, h = get_resolution(filename)
-    
-    command = ['ffmpeg',
-            '-i', filename,
-            '-f', 'image2pipe',
-            '-pix_fmt', 'rgb24',
-            '-vsync', '0',
-            '-vcodec', 'rawvideo', '-']
-    
-    i = 0
-    with sp.Popen(command, stdout = sp.PIPE, bufsize=-1) as pipe:
-        while True:
-            data = pipe.stdout.read(w*h*3)
-            if not data:
-                break
-            i += 1
-            if i > limit and limit != -1:
-                continue
-            if i > skip:
-                yield np.frombuffer(data, dtype='uint8').reshape((h, w, 3))
-            
-                
-                
-    
-def downsample_tensor(X, factor):
-    length = X.shape[0]//factor * factor
-    return np.mean(X[:length].reshape(-1, factor, *X.shape[1:]), axis=1)
-
-def render_animation(keypoints, keypoints_metadata, poses, skeleton, fps, bitrate, azim, output, viewport,
-                     limit=-1, downsample=1, size=6, input_video_path=None, input_video_skip=0):
-    """
-    TODO
-    Render an animation. The supported output modes are:
-     -- 'interactive': display an interactive figure
-                       (also works on notebooks if associated with %matplotlib inline)
-     -- 'html': render the animation as HTML5 video. Can be displayed in a notebook using HTML(...).
-     -- 'filename.mp4': render and export the animation as an h264 video (requires ffmpeg).
-     -- 'filename.gif': render and export the animation a gif file (requires imagemagick).
-    """
-    plt.ioff()
-    fig = plt.figure(figsize=(size*(1 + len(poses)), size))
-    ax_in = fig.add_subplot(1, 1 + len(poses), 1)
-    ax_in.get_xaxis().set_visible(False)
-    ax_in.get_yaxis().set_visible(False)
-    ax_in.set_axis_off()
-    ax_in.set_title('Input')
-
-    ax_3d = []
-    lines_3d = []
-    trajectories = []
-    radius = 1.7
-    for index, (title, data) in enumerate(poses.items()):
-        ax = fig.add_subplot(1, 1 + len(poses), index+2, projection='3d')
-        ax.view_init(elev=15., azim=azim)
-        ax.set_xlim3d([-radius/2, radius/2])
-        ax.set_zlim3d([0, radius])
-        ax.set_ylim3d([-radius/2, radius/2])
-        try:
-            ax.set_aspect('equal')
-        except NotImplementedError:
-            ax.set_aspect('auto')
-        ax.set_xticklabels([])
-        ax.set_yticklabels([])
-        ax.set_zticklabels([])
-        ax.dist = 7.5
-        ax.set_title(title) #, pad=35
-        ax_3d.append(ax)
-        lines_3d.append([])
-        trajectories.append(data[:, 0, [0, 1]])
-    poses = list(poses.values())
-
-    # Decode video
-    if input_video_path is None:
-        # Black background
-        all_frames = np.zeros((keypoints.shape[0], viewport[1], viewport[0]), dtype='uint8')
-    else:
-        # Load video using ffmpeg
-        all_frames = []
-        for f in read_video(input_video_path, skip=input_video_skip, limit=limit):
-            all_frames.append(f)
-        effective_length = min(keypoints.shape[0], len(all_frames))
-        all_frames = all_frames[:effective_length]
-        
-        keypoints = keypoints[input_video_skip:] # todo remove
-        for idx in range(len(poses)):
-            poses[idx] = poses[idx][input_video_skip:]
-        
-        if fps is None:
-            fps = get_fps(input_video_path)
-    
-    if downsample > 1:
-        keypoints = downsample_tensor(keypoints, downsample)
-        all_frames = downsample_tensor(np.array(all_frames), downsample).astype('uint8')
-        for idx in range(len(poses)):
-            poses[idx] = downsample_tensor(poses[idx], downsample)
-            trajectories[idx] = downsample_tensor(trajectories[idx], downsample)
-        fps /= downsample
-
-    initialized = False
-    image = None
-    lines = []
-    points = None
-    
-    if limit < 1:
-        limit = len(all_frames)
-    else:
-        limit = min(limit, len(all_frames))
-
-    parents = skeleton.parents()
-    def update_video(i):
-        nonlocal initialized, image, lines, points
-
-        for n, ax in enumerate(ax_3d):
-            ax.set_xlim3d([-radius/2 + trajectories[n][i, 0], radius/2 + trajectories[n][i, 0]])
-            ax.set_ylim3d([-radius/2 + trajectories[n][i, 1], radius/2 + trajectories[n][i, 1]])
-
-        # Update 2D poses
-        joints_right_2d = keypoints_metadata['keypoints_symmetry'][1]
-        colors_2d = np.full(keypoints.shape[1], 'black')
-        colors_2d[joints_right_2d] = 'red'
-        if not initialized:
-            image = ax_in.imshow(all_frames[i], aspect='equal')
-            
-            for j, j_parent in enumerate(parents):
-                if j_parent == -1:
-                    continue
-                    
-                if len(parents) == keypoints.shape[1] and keypoints_metadata['layout_name'] != 'coco':
-                    # Draw skeleton only if keypoints match (otherwise we don't have the parents definition)
-                    lines.append(ax_in.plot([keypoints[i, j, 0], keypoints[i, j_parent, 0]],
-                                            [keypoints[i, j, 1], keypoints[i, j_parent, 1]], color='pink'))
-
-                col = 'red' if j in skeleton.joints_right() else 'black'
-                for n, ax in enumerate(ax_3d):
-                    pos = poses[n][i]
-                    lines_3d[n].append(ax.plot([pos[j, 0], pos[j_parent, 0]],
-                                               [pos[j, 1], pos[j_parent, 1]],
-                                               [pos[j, 2], pos[j_parent, 2]], zdir='z', c=col))
-
-            points = ax_in.scatter(*keypoints[i].T, 10, color=colors_2d, edgecolors='white', zorder=10)
-
-            initialized = True
-        else:
-            image.set_data(all_frames[i])
-
-            for j, j_parent in enumerate(parents):
-                if j_parent == -1:
-                    continue
-                
-                if len(parents) == keypoints.shape[1] and keypoints_metadata['layout_name'] != 'coco':
-                    lines[j-1][0].set_data([keypoints[i, j, 0], keypoints[i, j_parent, 0]],
-                                           [keypoints[i, j, 1], keypoints[i, j_parent, 1]])
-
-                for n, ax in enumerate(ax_3d):
-                    pos = poses[n][i]
-                    lines_3d[n][j-1][0].set_xdata(np.array([pos[j, 0], pos[j_parent, 0]]))
-                    lines_3d[n][j-1][0].set_ydata(np.array([pos[j, 1], pos[j_parent, 1]]))
-                    lines_3d[n][j-1][0].set_3d_properties(np.array([pos[j, 2], pos[j_parent, 2]]), zdir='z')
-
-            points.set_offsets(keypoints[i])
-        
-        print('{}/{}      '.format(i, limit), end='\r')
-        
-
-    fig.tight_layout()
-    
-    anim = FuncAnimation(fig, update_video, frames=np.arange(0, limit), interval=1000/fps, repeat=False)
-    if output.endswith('.mp4'):
-        Writer = writers['ffmpeg']
-        writer = Writer(fps=fps, metadata={}, bitrate=bitrate)
-        anim.save(output, writer=writer)
-    elif output.endswith('.gif'):
-        anim.save(output, dpi=80, writer='imagemagick')
-    else:
-        raise ValueError('Unsupported output format (only .mp4 and .gif are supported)')
-    plt.close()

VideoPose3D/data/ConvertHumanEva.m

@@ -1,157 +0,0 @@
-% Copyright (c) 2018-present, Facebook, Inc.
-% All rights reserved.
-%
-% This source code is licensed under the license found in the
-% LICENSE file in the root directory of this source tree.
-%
-
-function [] = ConvertDataset()
-
-    N_JOINTS = 15; % Set to 20 if you want to export a 20-joint skeleton
-
-    function [pose_out] = ExtractPose15(pose, dimensions)
-        % We use the same 15-joint skeleton as in the evaluation
-        % script "@body_pose/error.m". Proximal and Distal joints
-        % are averaged.
-        pose_out = NaN(15, dimensions);
-        pose_out(1, :) = pose.torsoDistal; % Pelvis (root)
-        pose_out(2, :) = (pose.torsoProximal + pose.headProximal) / 2; % Thorax
-        pose_out(3, :) = pose.upperLArmProximal; % Left shoulder
-        pose_out(4, :) = (pose.upperLArmDistal + pose.lowerLArmProximal) / 2; % Left elbow
-        pose_out(5, :) = pose.lowerLArmDistal; % Left wrist
-        pose_out(6, :) = pose.upperRArmProximal; % Right shoulder 
-        pose_out(7, :) = (pose.upperRArmDistal + pose.lowerRArmProximal) / 2; % Right elbow
-        pose_out(8, :) = pose.lowerRArmDistal; % Right wrist
-        pose_out(9, :) = pose.upperLLegProximal; % Left hip
-        pose_out(10, :) = (pose.upperLLegDistal + pose.lowerLLegProximal) / 2; % Left knee
-        pose_out(11, :) = pose.lowerLLegDistal; % Left ankle 
-        pose_out(12, :) = pose.upperRLegProximal; % Right hip
-        pose_out(13, :) = (pose.upperRLegDistal + pose.lowerRLegProximal) / 2; % Right knee
-        pose_out(14, :) = pose.lowerRLegDistal; % Right ankle 
-        pose_out(15, :) = pose.headDistal; % Head
-    end
-
-    function [pose_out] = ExtractPose20(pose, dimensions)
-        pose_out = NaN(20, dimensions);
-        pose_out(1, :) = pose.torsoDistal; % Pelvis (root)
-        pose_out(2, :) = pose.torsoProximal;
-        pose_out(3, :) = pose.headProximal;
-        pose_out(4, :) = pose.upperLArmProximal; % Left shoulder
-        pose_out(5, :) = pose.upperLArmDistal;
-        pose_out(6, :) = pose.lowerLArmProximal;
-        pose_out(7, :) = pose.lowerLArmDistal; % Left wrist
-        pose_out(8, :) = pose.upperRArmProximal; % Right shoulder 
-        pose_out(9, :) = pose.upperRArmDistal;
-        pose_out(10, :) = pose.lowerRArmProximal;
-        pose_out(11, :) = pose.lowerRArmDistal; % Right wrist
-        pose_out(12, :) = pose.upperLLegProximal; % Left hip
-        pose_out(13, :) = pose.upperLLegDistal;
-        pose_out(14, :) = pose.lowerLLegProximal;
-        pose_out(15, :) = pose.lowerLLegDistal; % Left ankle 
-        pose_out(16, :) = pose.upperRLegProximal; % Right hip
-        pose_out(17, :) = pose.upperRLegDistal;
-        pose_out(18, :) = pose.lowerRLegProximal;
-        pose_out(19, :) = pose.lowerRLegDistal; % Right ankle 
-        pose_out(20, :) = pose.headDistal; % Head
-    end
-
-    addpath('./TOOLBOX_calib/');
-    addpath('./TOOLBOX_common/');
-    addpath('./TOOLBOX_dxAvi/');
-    addpath('./TOOLBOX_readc3d/'); 
-
-    % Create the output directory for the converted dataset
-    OUT_DIR = ['./converted_', int2str(N_JOINTS), 'j'];
-    warning('off', 'MATLAB:MKDIR:DirectoryExists');
-    mkdir(OUT_DIR);
-
-    % We use the validation set as the test set
-    for SPLIT = {'Train', 'Validate'}
-        mkdir([OUT_DIR, '/', SPLIT{1}]);
-        CurrentDataset = he_dataset('HumanEvaI', SPLIT{1});
-
-        for SEQ = 1:length(CurrentDataset)
-
-            Subject = char(get(CurrentDataset(SEQ), 'SubjectName'));
-            Action = char(get(CurrentDataset(SEQ), 'ActionType'));
-            Trial = char(get(CurrentDataset(SEQ), 'Trial'));
-            DatasetBasePath = char(get(CurrentDataset(SEQ), 'DatasetBasePath'));
-            if Trial ~= '1'
-                % We are only interested in fully-annotated data
-                continue;
-            end
-
-            if strcmp(Action, 'ThrowCatch') && strcmp(Subject, 'S3')
-                % Damaged mocap stream
-                continue;
-            end
-
-            fprintf('Converting...\n')
-            fprintf('\tSplit: %s\n', SPLIT{1});
-            fprintf('\tSubject: %s\n', Subject);
-            fprintf('\tAction: %s\n', Action);    
-            fprintf('\tTrial: %s\n', Trial);
-
-            % Create subject directory if it does not exist
-            mkdir([OUT_DIR, '/', SPLIT{1}, '/', Subject]);
-
-            % Load the sequence
-            [~, ~, MocapStream, MocapStream_Enabled] ...
-                                    = sync_stream(CurrentDataset(SEQ));
-
-            % Set frame range
-            FrameStart = get(CurrentDataset(SEQ), 'FrameStart');
-            FrameStart = [FrameStart{:}];
-            FrameEnd   = get(CurrentDataset(SEQ), 'FrameEnd');    
-            FrameEnd   = [FrameEnd{:}];
-
-            fprintf('\tNum. frames: %d\n', FrameEnd - FrameStart + 1);
-            poses_3d = NaN(FrameEnd - FrameStart + 1, N_JOINTS, 3);
-            poses_2d = NaN(3, FrameEnd - FrameStart + 1, N_JOINTS, 2);
-            corrupt = 0;
-            for FRAME = FrameStart:FrameEnd
-
-                if (MocapStream_Enabled)
-                    [MocapStream, pose, ValidPose] = cur_frame(MocapStream, FRAME, 'body_pose');
-
-                    if (ValidPose)
-                        i = FRAME - FrameStart + 1;
-                        
-                        % Extract 3D pose
-                        if N_JOINTS == 15
-                            poses_3d(i, :, :) = ExtractPose15(pose, 3);
-                        else
-                            poses_3d(i, :, :) = ExtractPose20(pose, 3);
-                        end
-                        
-                        % Extract ground-truth 2D pose via camera
-                        % projection
-                        for CAM = 1:3
-                            if (CAM == 1)
-                                CameraName = 'C1';
-                            elseif (CAM == 2)
-                                CameraName = 'C2';    
-                            elseif (CAM == 3)
-                                CameraName = 'C3';
-                            end
-                            CalibrationFilename = [DatasetBasePath, Subject, '/Calibration_Data/', CameraName,  '.cal'];
-                            pose_2d = project2d(pose, CalibrationFilename);
-                            if N_JOINTS == 15
-                                poses_2d(CAM, i, :, :) = ExtractPose15(pose_2d, 2);
-                            else
-                                poses_2d(CAM, i, :, :) = ExtractPose20(pose_2d, 2);
-                            end
-                        end
-                        
-                    else
-                        corrupt = corrupt + 1;
-                    end
-                end
-            end
-            fprintf('\n%d out of %d frames are damaged\n', corrupt, FrameEnd - FrameStart + 1);
-            FileName = [OUT_DIR, '/', SPLIT{1}, '/', Subject, '/', Action, '_', Trial, '.mat'];
-            save(FileName, 'poses_3d', 'poses_2d');
-            fprintf('... saved to %s\n\n', FileName);
-        end
-    end
-end

VideoPose3D/data/convert_cdf_to_mat.m

@@ -1,20 +0,0 @@
-% Copyright (c) 2018-present, Facebook, Inc.
-% All rights reserved.
-%
-% This source code is licensed under the license found in the
-% LICENSE file in the root directory of this source tree.
-%
-
-% Extract "Poses_D3_Positions_S*.tgz" to the "pose" directory
-% and run this script to convert all .cdf files to .mat
-
-pose_directory = 'pose';
-dirs = dir(strcat(pose_directory, '/*/MyPoseFeatures/D3_Positions/*.cdf'));
-
-paths = {dirs.folder};
-names = {dirs.name};
-
-for i = 1:numel(names)
-    data = cdfread(strcat(paths{i}, '/', names{i}));
-    save(strcat(paths{i}, '/', names{i}, '.mat'), 'data');
-end

VideoPose3D/data/data_utils.py

@@ -1,102 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-
-mpii_metadata = {
-    'layout_name': 'mpii',
-    'num_joints': 16,
-    'keypoints_symmetry': [
-        [3, 4, 5, 13, 14, 15],
-        [0, 1, 2, 10, 11, 12],
-    ]
-}
-
-coco_metadata = {
-    'layout_name': 'coco',
-    'num_joints': 17,
-    'keypoints_symmetry': [
-        [1, 3, 5, 7, 9, 11, 13, 15],
-        [2, 4, 6, 8, 10, 12, 14, 16],
-    ]
-}
-
-h36m_metadata = {
-    'layout_name': 'h36m',
-    'num_joints': 17,
-    'keypoints_symmetry': [
-        [4, 5, 6, 11, 12, 13],
-        [1, 2, 3, 14, 15, 16],
-    ]
-}
-
-humaneva15_metadata = {
-    'layout_name': 'humaneva15',
-    'num_joints': 15,
-    'keypoints_symmetry': [
-        [2, 3, 4, 8, 9, 10],
-        [5, 6, 7, 11, 12, 13]
-    ]
-}
-
-humaneva20_metadata = {
-    'layout_name': 'humaneva20',
-    'num_joints': 20,
-    'keypoints_symmetry': [
-        [3, 4, 5, 6, 11, 12, 13, 14],
-        [7, 8, 9, 10, 15, 16, 17, 18]
-    ]
-}
-
-def suggest_metadata(name):
-    names = []
-    for metadata in [mpii_metadata, coco_metadata, h36m_metadata, humaneva15_metadata, humaneva20_metadata]:
-        if metadata['layout_name'] in name:
-            return metadata
-        names.append(metadata['layout_name'])
-    raise KeyError('Cannot infer keypoint layout from name "{}". Tried {}.'.format(name, names))
-
-def import_detectron_poses(path):
-    # Latin1 encoding because Detectron runs on Python 2.7
-    data = np.load(path, encoding='latin1')
-    kp = data['keypoints']
-    bb = data['boxes']
-    results = []
-    for i in range(len(bb)):
-        if len(bb[i][1]) == 0:
-            assert i > 0
-            # Use last pose in case of detection failure
-            results.append(results[-1])
-            continue
-        best_match = np.argmax(bb[i][1][:, 4])
-        keypoints = kp[i][1][best_match].T.copy()
-        results.append(keypoints)
-    results = np.array(results)
-    return results[:, :, 4:6] # Soft-argmax
-    #return results[:, :, [0, 1, 3]] # Argmax + score
-    
-    
-def import_cpn_poses(path):
-    data = np.load(path)
-    kp = data['keypoints']
-    return kp[:, :, :2]
-    
-    
-def import_sh_poses(path):
-    import h5py
-    with h5py.File(path) as hf:
-        positions = hf['poses'].value
-    return positions.astype('float32')
-    
-def suggest_pose_importer(name):
-    if 'detectron' in name:
-        return import_detectron_poses
-    if 'cpn' in name:
-        return import_cpn_poses
-    if 'sh' in name:
-        return import_sh_poses
-    raise KeyError('Cannot infer keypoint format from name "{}". Tried detectron, cpn, sh.'.format(name))

VideoPose3D/data/prepare_data_2d_custom.py

@@ -1,98 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-from glob import glob
-import os
-import sys
-
-import argparse
-from data_utils import suggest_metadata
-
-output_prefix_2d = 'data_2d_custom_'
-
-def decode(filename):
-    # Latin1 encoding because Detectron runs on Python 2.7
-    print('Processing {}'.format(filename))
-    data = np.load(filename, encoding='latin1', allow_pickle=True)
-    bb = data['boxes']
-    kp = data['keypoints']
-    metadata = data['metadata'].item()
-    results_bb = []
-    results_kp = []
-    for i in range(len(bb)):
-        if len(bb[i][1]) == 0 or len(kp[i][1]) == 0:
-            # No bbox/keypoints detected for this frame -> will be interpolated
-            results_bb.append(np.full(4, np.nan, dtype=np.float32)) # 4 bounding box coordinates
-            results_kp.append(np.full((17, 4), np.nan, dtype=np.float32)) # 17 COCO keypoints
-            continue
-        best_match = np.argmax(bb[i][1][:, 4])
-        best_bb = bb[i][1][best_match, :4]
-        best_kp = kp[i][1][best_match].T.copy()
-        results_bb.append(best_bb)
-        results_kp.append(best_kp)
-        
-    bb = np.array(results_bb, dtype=np.float32)
-    kp = np.array(results_kp, dtype=np.float32)
-    kp = kp[:, :, :2] # Extract (x, y)
-    
-    # Fix missing bboxes/keypoints by linear interpolation
-    mask = ~np.isnan(bb[:, 0])
-    indices = np.arange(len(bb))
-    for i in range(4):
-        bb[:, i] = np.interp(indices, indices[mask], bb[mask, i])
-    for i in range(17):
-        for j in range(2):
-            kp[:, i, j] = np.interp(indices, indices[mask], kp[mask, i, j])
-    
-    print('{} total frames processed'.format(len(bb)))
-    print('{} frames were interpolated'.format(np.sum(~mask)))
-    print('----------')
-    
-    return [{
-        'start_frame': 0, # Inclusive
-        'end_frame': len(kp), # Exclusive
-        'bounding_boxes': bb,
-        'keypoints': kp,
-    }], metadata
-
-
-if __name__ == '__main__':
-    if os.path.basename(os.getcwd()) != 'data':
-        print('This script must be launched from the "data" directory')
-        exit(0)
-        
-    parser = argparse.ArgumentParser(description='Custom dataset creator')
-    parser.add_argument('-i', '--input', type=str, default='', metavar='PATH', help='detections directory')
-    parser.add_argument('-o', '--output', type=str, default='', metavar='PATH', help='output suffix for 2D detections')
-    args = parser.parse_args()
-    
-    if not args.input:
-        print('Please specify the input directory')
-        exit(0)
-        
-    if not args.output:
-        print('Please specify an output suffix (e.g. detectron_pt_coco)')
-        exit(0)
-    
-    print('Parsing 2D detections from', args.input)
-    
-    metadata = suggest_metadata('coco')
-    metadata['video_metadata'] = {}
-    
-    output = {}
-    file_list = glob(args.input + '/*.npz')
-    for f in file_list:
-        canonical_name = os.path.splitext(os.path.basename(f))[0]
-        data, video_metadata = decode(f)
-        output[canonical_name] = {}
-        output[canonical_name]['custom'] = [data[0]['keypoints'].astype('float32')]
-        metadata['video_metadata'][canonical_name] = video_metadata
-
-    print('Saving...')
-    np.savez_compressed(output_prefix_2d + args.output, positions_2d=output, metadata=metadata)
-    print('Done.')

VideoPose3D/data/prepare_data_2d_h36m_generic.py

@@ -1,89 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import argparse
-import os
-import zipfile
-import numpy as np
-import h5py
-import re
-from glob import glob
-from shutil import rmtree
-from data_utils import suggest_metadata, suggest_pose_importer
-
-import sys
-sys.path.append('../')
-from common.utils import wrap
-from itertools import groupby
-
-output_prefix_2d = 'data_2d_h36m_'
-cam_map = {
-    '54138969': 0,
-    '55011271': 1,
-    '58860488': 2,
-    '60457274': 3,
-}
-
-if __name__ == '__main__':
-    if os.path.basename(os.getcwd()) != 'data':
-        print('This script must be launched from the "data" directory')
-        exit(0)
-        
-    parser = argparse.ArgumentParser(description='Human3.6M dataset converter')
-    
-    parser.add_argument('-i', '--input', default='', type=str, metavar='PATH', help='input path to 2D detections')
-    parser.add_argument('-o', '--output', default='', type=str, metavar='PATH', help='output suffix for 2D detections (e.g. detectron_pt_coco)')
-    
-    args = parser.parse_args()
-        
-    if not args.input:
-        print('Please specify the input directory')
-        exit(0)
-        
-    if not args.output:
-        print('Please specify an output suffix (e.g. detectron_pt_coco)')
-        exit(0)
-
-    import_func = suggest_pose_importer(args.output)
-    metadata = suggest_metadata(args.output)
-
-    print('Parsing 2D detections from', args.input)
-
-    output = {}
-    file_list = glob(args.input + '/S*/*.mp4.npz')
-    for f in file_list:
-        path, fname = os.path.split(f)
-        subject = os.path.basename(path)
-        assert subject.startswith('S'), subject + ' does not look like a subject directory'
-
-        if '_ALL' in fname:
-            continue
-        
-        m = re.search('(.*)\\.([0-9]+)\\.mp4\\.npz', fname)
-        action = m.group(1)
-        camera = m.group(2)
-        camera_idx = cam_map[camera]
-        
-        if subject == 'S11' and action == 'Directions':
-            continue # Discard corrupted video
-        
-        # Use consistent naming convention
-        canonical_name = action.replace('TakingPhoto', 'Photo') \
-                               .replace('WalkingDog', 'WalkDog')
-
-        keypoints = import_func(f)
-        assert keypoints.shape[1] == metadata['num_joints']
-        
-        if subject not in output:
-            output[subject] = {}
-        if canonical_name not in output[subject]:
-            output[subject][canonical_name] = [None, None, None, None]
-        output[subject][canonical_name][camera_idx] = keypoints.astype('float32')
-
-    print('Saving...')
-    np.savez_compressed(output_prefix_2d + args.output, positions_2d=output, metadata=metadata)
-    print('Done.')

VideoPose3D/data/prepare_data_2d_h36m_sh.py

@@ -1,112 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import argparse
-import os
-import zipfile
-import tarfile
-import numpy as np
-import h5py
-from glob import glob
-from shutil import rmtree
-
-import sys
-sys.path.append('../')
-from common.h36m_dataset import Human36mDataset
-from common.camera import world_to_camera, project_to_2d, image_coordinates
-from common.utils import wrap
-
-output_filename_pt = 'data_2d_h36m_sh_pt_mpii'
-output_filename_ft = 'data_2d_h36m_sh_ft_h36m'
-subjects = ['S1', 'S5', 'S6', 'S7', 'S8', 'S9', 'S11']
-cam_map = {
-    '54138969': 0,
-    '55011271': 1,
-    '58860488': 2,
-    '60457274': 3,
-}
-
-metadata = {
-    'num_joints': 16,
-    'keypoints_symmetry': [
-        [3, 4, 5, 13, 14, 15],
-        [0, 1, 2, 10, 11, 12],
-    ]
-}
-
-def process_subject(subject, file_list, output):
-    if subject == 'S11':
-        assert len(file_list) == 119, "Expected 119 files for subject " + subject + ", got " + str(len(file_list))
-    else:
-        assert len(file_list) == 120, "Expected 120 files for subject " + subject + ", got " + str(len(file_list))
-        
-    for f in file_list:
-        action, cam = os.path.splitext(os.path.basename(f))[0].replace('_', ' ').split('.')
-        
-        if subject == 'S11' and action == 'Directions':
-            continue # Discard corrupted video
-        
-        if action not in output[subject]:
-            output[subject][action] = [None, None, None, None]
-        
-        with h5py.File(f) as hf:
-            positions = hf['poses'].value
-            output[subject][action][cam_map[cam]] = positions.astype('float32')    
-
-if __name__ == '__main__':
-    if os.path.basename(os.getcwd()) != 'data':
-        print('This script must be launched from the "data" directory')
-        exit(0)
-        
-    parser = argparse.ArgumentParser(description='Human3.6M dataset downloader/converter')
-    
-    parser.add_argument('-pt', '--pretrained', default='', type=str, metavar='PATH', help='convert pretrained dataset')
-    parser.add_argument('-ft', '--fine-tuned', default='', type=str, metavar='PATH', help='convert fine-tuned dataset')
-    
-    args = parser.parse_args()
-        
-    if args.pretrained:
-        print('Converting pretrained dataset from', args.pretrained)
-        print('Extracting...')
-        with zipfile.ZipFile(args.pretrained, 'r') as archive:
-            archive.extractall('sh_pt')
-        
-        print('Converting...')
-        output = {}
-        for subject in subjects:
-            output[subject] = {}
-            file_list = glob('sh_pt/h36m/' + subject + '/StackedHourglass/*.h5')
-            process_subject(subject, file_list, output)
-        
-        print('Saving...')
-        np.savez_compressed(output_filename_pt, positions_2d=output, metadata=metadata)
-        
-        print('Cleaning up...')
-        rmtree('sh_pt')
-        
-        print('Done.')
-        
-    if args.fine_tuned:
-        print('Converting fine-tuned dataset from', args.fine_tuned)
-        print('Extracting...')
-        with tarfile.open(args.fine_tuned, 'r:gz') as archive:
-            archive.extractall('sh_ft')
-        
-        print('Converting...')
-        output = {}
-        for subject in subjects:
-            output[subject] = {}
-            file_list = glob('sh_ft/' + subject + '/StackedHourglassFineTuned240/*.h5')
-            process_subject(subject, file_list, output)
-        
-        print('Saving...')
-        np.savez_compressed(output_filename_ft, positions_2d=output, metadata=metadata)
-        
-        print('Cleaning up...')
-        rmtree('sh_ft')
-        
-        print('Done.')

VideoPose3D/data/prepare_data_h36m.py

@@ -1,173 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import argparse
-import os
-import zipfile
-import numpy as np
-import h5py
-from glob import glob
-from shutil import rmtree
-
-import sys
-sys.path.append('../')
-from common.h36m_dataset import Human36mDataset
-from common.camera import world_to_camera, project_to_2d, image_coordinates
-from common.utils import wrap
-
-output_filename = 'data_3d_h36m'
-output_filename_2d = 'data_2d_h36m_gt'
-subjects = ['S1', 'S5', 'S6', 'S7', 'S8', 'S9', 'S11']
-
-if __name__ == '__main__':
-    if os.path.basename(os.getcwd()) != 'data':
-        print('This script must be launched from the "data" directory')
-        exit(0)
-        
-    parser = argparse.ArgumentParser(description='Human3.6M dataset downloader/converter')
-    
-    # Convert dataset preprocessed by Martinez et al. in https://github.com/una-dinosauria/3d-pose-baseline
-    parser.add_argument('--from-archive', default='', type=str, metavar='PATH', help='convert preprocessed dataset')
-    
-    # Convert dataset from original source, using files converted to .mat (the Human3.6M dataset path must be specified manually)
-    # This option requires MATLAB to convert files using the provided script
-    parser.add_argument('--from-source', default='', type=str, metavar='PATH', help='convert original dataset')
-    
-    # Convert dataset from original source, using original .cdf files (the Human3.6M dataset path must be specified manually)
-    # This option does not require MATLAB, but the Python library cdflib must be installed
-    parser.add_argument('--from-source-cdf', default='', type=str, metavar='PATH', help='convert original dataset')
-    
-    args = parser.parse_args()
-    
-    if args.from_archive and args.from_source:
-        print('Please specify only one argument')
-        exit(0)
-    
-    if os.path.exists(output_filename + '.npz'):
-        print('The dataset already exists at', output_filename + '.npz')
-        exit(0)
-        
-    if args.from_archive:
-        print('Extracting Human3.6M dataset from', args.from_archive)
-        with zipfile.ZipFile(args.from_archive, 'r') as archive:
-            archive.extractall()
-        
-        print('Converting...')
-        output = {}
-        for subject in subjects:
-            output[subject] = {}
-            file_list = glob('h36m/' + subject + '/MyPoses/3D_positions/*.h5')
-            assert len(file_list) == 30, "Expected 30 files for subject " + subject + ", got " + str(len(file_list))
-            for f in file_list:
-                action = os.path.splitext(os.path.basename(f))[0]
-                
-                if subject == 'S11' and action == 'Directions':
-                    continue # Discard corrupted video
-                
-                with h5py.File(f) as hf:
-                    positions = hf['3D_positions'].value.reshape(32, 3, -1).transpose(2, 0, 1)
-                    positions /= 1000 # Meters instead of millimeters
-                    output[subject][action] = positions.astype('float32')
-        
-        print('Saving...')
-        np.savez_compressed(output_filename, positions_3d=output)
-        
-        print('Cleaning up...')
-        rmtree('h36m')
-        
-        print('Done.')
-                
-    elif args.from_source:
-        print('Converting original Human3.6M dataset from', args.from_source)
-        output = {}
-        
-        from scipy.io import loadmat
-        
-        for subject in subjects:
-            output[subject] = {}
-            file_list = glob(args.from_source + '/' + subject + '/MyPoseFeatures/D3_Positions/*.cdf.mat')
-            assert len(file_list) == 30, "Expected 30 files for subject " + subject + ", got " + str(len(file_list))
-            for f in file_list:
-                action = os.path.splitext(os.path.splitext(os.path.basename(f))[0])[0]
-                
-                if subject == 'S11' and action == 'Directions':
-                    continue # Discard corrupted video
-                    
-                # Use consistent naming convention
-                canonical_name = action.replace('TakingPhoto', 'Photo') \
-                                       .replace('WalkingDog', 'WalkDog')
-                
-                hf = loadmat(f)
-                positions = hf['data'][0, 0].reshape(-1, 32, 3)
-                positions /= 1000 # Meters instead of millimeters
-                output[subject][canonical_name] = positions.astype('float32')
-        
-        print('Saving...')
-        np.savez_compressed(output_filename, positions_3d=output)
-        
-        print('Done.')
-        
-    elif args.from_source_cdf:
-        print('Converting original Human3.6M dataset from', args.from_source_cdf, '(CDF files)')
-        output = {}
-        
-        import cdflib
-        
-        for subject in subjects:
-            output[subject] = {}
-            file_list = glob(args.from_source_cdf + '/' + subject + '/MyPoseFeatures/D3_Positions/*.cdf')
-            assert len(file_list) == 30, "Expected 30 files for subject " + subject + ", got " + str(len(file_list))
-            for f in file_list:
-                action = os.path.splitext(os.path.basename(f))[0]
-                
-                if subject == 'S11' and action == 'Directions':
-                    continue # Discard corrupted video
-                    
-                # Use consistent naming convention
-                canonical_name = action.replace('TakingPhoto', 'Photo') \
-                                       .replace('WalkingDog', 'WalkDog')
-                
-                hf = cdflib.CDF(f)
-                positions = hf['Pose'].reshape(-1, 32, 3)
-                positions /= 1000 # Meters instead of millimeters
-                output[subject][canonical_name] = positions.astype('float32')
-        
-        print('Saving...')
-        np.savez_compressed(output_filename, positions_3d=output)
-        
-        print('Done.')
-            
-    else:
-        print('Please specify the dataset source')
-        exit(0)
-        
-    # Create 2D pose file
-    print('')
-    print('Computing ground-truth 2D poses...')
-    dataset = Human36mDataset(output_filename + '.npz')
-    output_2d_poses = {}
-    for subject in dataset.subjects():
-        output_2d_poses[subject] = {}
-        for action in dataset[subject].keys():
-            anim = dataset[subject][action]
-            
-            positions_2d = []
-            for cam in anim['cameras']:
-                pos_3d = world_to_camera(anim['positions'], R=cam['orientation'], t=cam['translation'])
-                pos_2d = wrap(project_to_2d, pos_3d, cam['intrinsic'], unsqueeze=True)
-                pos_2d_pixel_space = image_coordinates(pos_2d, w=cam['res_w'], h=cam['res_h'])
-                positions_2d.append(pos_2d_pixel_space.astype('float32'))
-            output_2d_poses[subject][action] = positions_2d
-            
-    print('Saving...')
-    metadata = {
-        'num_joints': dataset.skeleton().num_joints(),
-        'keypoints_symmetry': [dataset.skeleton().joints_left(), dataset.skeleton().joints_right()]
-    }
-    np.savez_compressed(output_filename_2d, positions_2d=output_2d_poses, metadata=metadata)
-    
-    print('Done.')

VideoPose3D/data/prepare_data_humaneva.py

@@ -1,246 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import argparse
-import os
-import zipfile
-import numpy as np
-import h5py
-import re
-from glob import glob
-from shutil import rmtree
-from data_utils import suggest_metadata, suggest_pose_importer
-
-import sys
-sys.path.append('../')
-from common.utils import wrap
-from itertools import groupby
-
-subjects = ['Train/S1', 'Train/S2', 'Train/S3', 'Validate/S1', 'Validate/S2', 'Validate/S3']
-
-cam_map = {
-    'C1': 0,
-    'C2': 1,
-    'C3': 2,
-}
-
-# Frame numbers for train/test split
-# format: [start_frame, end_frame[ (inclusive, exclusive)
-index = {
-    'Train/S1': {
-        'Walking 1': (590, 1203),
-        'Jog 1': (367, 740),
-        'ThrowCatch 1': (473, 945),
-        'Gestures 1': (395, 801),
-        'Box 1': (385, 789),
-    },
-    'Train/S2': {
-        'Walking 1': (438, 876),
-        'Jog 1': (398, 795),
-        'ThrowCatch 1': (550, 1128),
-        'Gestures 1': (500, 901),
-        'Box 1': (382, 734),
-    },
-    'Train/S3': {
-        'Walking 1': (448, 939),
-        'Jog 1': (401, 842),
-        'ThrowCatch 1': (493, 1027),
-        'Gestures 1': (533, 1102),
-        'Box 1': (512, 1021),
-    },
-    'Validate/S1': {
-        'Walking 1': (5, 590),
-        'Jog 1': (5, 367),
-        'ThrowCatch 1': (5, 473),
-        'Gestures 1': (5, 395),
-        'Box 1': (5, 385),
-    },
-    'Validate/S2': {
-        'Walking 1': (5, 438),
-        'Jog 1': (5, 398),
-        'ThrowCatch 1': (5, 550),
-        'Gestures 1': (5, 500),
-        'Box 1': (5, 382),
-    },
-    'Validate/S3': {
-        'Walking 1': (5, 448),
-        'Jog 1': (5, 401),
-        'ThrowCatch 1': (5, 493),
-        'Gestures 1': (5, 533),
-        'Box 1': (5, 512),
-    },
-}
-
-# Frames to skip for each video (synchronization)
-sync_data = {
-    'S1': {
-        'Walking 1': (82, 81, 82),
-        'Jog 1': (51, 51, 50),
-        'ThrowCatch 1': (61, 61, 60),
-        'Gestures 1': (45, 45, 44),
-        'Box 1': (57, 57, 56),
-    },
-    'S2': {
-        'Walking 1': (115, 115, 114),
-        'Jog 1': (100, 100, 99),
-        'ThrowCatch 1': (127, 127, 127),
-        'Gestures 1': (122, 122, 121),
-        'Box 1': (119, 119, 117),
-    },
-    'S3': {
-        'Walking 1': (80, 80, 80),
-        'Jog 1': (65, 65, 65),
-        'ThrowCatch 1': (79, 79, 79),
-        'Gestures 1': (83, 83, 82),
-        'Box 1': (1, 1, 1),
-    },
-    'S4': {}
-}
-
-if __name__ == '__main__':
-    if os.path.basename(os.getcwd()) != 'data':
-        print('This script must be launched from the "data" directory')
-        exit(0)
-        
-    parser = argparse.ArgumentParser(description='HumanEva dataset converter')
-    
-    parser.add_argument('-p', '--path', default='', type=str, metavar='PATH', help='path to the processed HumanEva dataset')
-    parser.add_argument('--convert-3d', action='store_true', help='convert 3D mocap data')
-    parser.add_argument('--convert-2d', default='', type=str, metavar='PATH', help='convert user-supplied 2D detections')
-    parser.add_argument('-o', '--output', default='', type=str, metavar='PATH', help='output suffix for 2D detections (e.g. detectron_pt_coco)')
-    
-    args = parser.parse_args()
-        
-    if not args.convert_2d and not args.convert_3d:
-        print('Please specify one conversion mode')
-        exit(0)
-        
- 
-    if args.path:
-        print('Parsing HumanEva dataset from', args.path)
-        output = {}
-        output_2d = {}
-        frame_mapping = {}
-        
-        from scipy.io import loadmat
-        
-        num_joints = None
-        
-        for subject in subjects:
-            output[subject] = {}
-            output_2d[subject] = {}
-            split, subject_name = subject.split('/')
-            if subject_name not in frame_mapping:
-                frame_mapping[subject_name] = {}
-            
-            file_list = glob(args.path + '/' + subject + '/*.mat')
-            for f in file_list:
-                action = os.path.splitext(os.path.basename(f))[0]
-                    
-                # Use consistent naming convention
-                canonical_name = action.replace('_', ' ')
-                
-                hf = loadmat(f)
-                positions = hf['poses_3d']
-                positions_2d = hf['poses_2d'].transpose(1, 0, 2, 3) # Ground-truth 2D poses
-                assert positions.shape[0] == positions_2d.shape[0] and positions.shape[1] == positions_2d.shape[2]
-                assert num_joints is None or num_joints == positions.shape[1], "Joint number inconsistency among files"
-                num_joints = positions.shape[1]
-                
-                # Sanity check for the sequence length
-                assert positions.shape[0] == index[subject][canonical_name][1] - index[subject][canonical_name][0]
-                
-                # Split corrupted motion capture streams into contiguous chunks
-                # e.g. 012XX567X9 is split into "012", "567", and "9".
-                all_chunks = [list(v) for k, v in groupby(positions, lambda x: np.isfinite(x).all())]
-                all_chunks_2d = [list(v) for k, v in groupby(positions_2d, lambda x: np.isfinite(x).all())]
-                assert len(all_chunks) == len(all_chunks_2d)
-                current_index = index[subject][canonical_name][0]
-                chunk_indices = []
-                for i, chunk in enumerate(all_chunks):
-                    next_index = current_index + len(chunk)
-                    name = canonical_name + ' chunk' + str(i)
-                    if np.isfinite(chunk).all():
-                        output[subject][name] = np.array(chunk, dtype='float32') / 1000
-                        output_2d[subject][name] = list(np.array(all_chunks_2d[i], dtype='float32').transpose(1, 0, 2, 3))
-                    chunk_indices.append((current_index, next_index, np.isfinite(chunk).all(), split, name))
-                    current_index = next_index
-                assert current_index == index[subject][canonical_name][1]
-                if canonical_name not in frame_mapping[subject_name]:
-                    frame_mapping[subject_name][canonical_name] = []
-                frame_mapping[subject_name][canonical_name] += chunk_indices
-        
-        metadata = suggest_metadata('humaneva' + str(num_joints))
-        output_filename = 'data_3d_' + metadata['layout_name']
-        output_prefix_2d = 'data_2d_' + metadata['layout_name'] + '_'
-        
-        if args.convert_3d:
-            print('Saving...')
-            np.savez_compressed(output_filename, positions_3d=output)
-            np.savez_compressed(output_prefix_2d + 'gt', positions_2d=output_2d, metadata=metadata)
-            print('Done.')
-        
-    else:
-        print('Please specify the dataset source')
-        exit(0)
-        
-    if args.convert_2d:
-        if not args.output:
-            print('Please specify an output suffix (e.g. detectron_pt_coco)')
-            exit(0)
-            
-        import_func = suggest_pose_importer(args.output)
-        metadata = suggest_metadata(args.output)
-            
-        print('Parsing 2D detections from', args.convert_2d)
-        
-        output = {}
-        file_list = glob(args.convert_2d + '/S*/*.avi.npz')
-        for f in file_list:
-            path, fname = os.path.split(f)
-            subject = os.path.basename(path)
-            assert subject.startswith('S'), subject + ' does not look like a subject directory'
-            
-            m = re.search('(.*) \\((.*)\\)', fname.replace('_', ' '))
-            action = m.group(1)
-            camera = m.group(2)
-            camera_idx = cam_map[camera]
-            
-            keypoints = import_func(f)
-            assert keypoints.shape[1] == metadata['num_joints']
-            
-            if action in sync_data[subject]:
-                sync_offset = sync_data[subject][action][camera_idx] - 1
-            else:
-                sync_offset = 0
-
-            if subject in frame_mapping and action in frame_mapping[subject]:
-                chunks = frame_mapping[subject][action]
-                for (start_idx, end_idx, labeled, split, name) in chunks:
-                    canonical_subject = split + '/' + subject
-                    if not labeled:
-                        canonical_subject = 'Unlabeled/' + canonical_subject
-                    if canonical_subject not in output:
-                        output[canonical_subject] = {}
-                    kps = keypoints[start_idx+sync_offset:end_idx+sync_offset]
-                    assert len(kps) == end_idx - start_idx, "Got len {}, expected {}".format(len(kps), end_idx - start_idx)
-                    
-                    if name not in output[canonical_subject]:
-                        output[canonical_subject][name] = [None, None, None]
-                    
-                    output[canonical_subject][name][camera_idx] = kps.astype('float32')
-            else:
-                canonical_subject = 'Unlabeled/' + subject
-                if canonical_subject not in output:
-                    output[canonical_subject] = {}
-                if action not in output[canonical_subject]:
-                        output[canonical_subject][action] = [None, None, None]
-                output[canonical_subject][action][camera_idx] = keypoints.astype('float32')
-                
-        print('Saving...')
-        np.savez_compressed(output_prefix_2d + args.output, positions_2d=output, metadata=metadata)
-        print('Done.')

VideoPose3D/inference/infer_video.py

@@ -1,100 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-"""Perform inference on a single video or all videos with a certain extension
-(e.g., .mp4) in a folder.
-"""
-
-from infer_simple import *
-import subprocess as sp
-import numpy as np
-
-def get_resolution(filename):
-    command = ['ffprobe', '-v', 'error', '-select_streams', 'v:0',
-               '-show_entries', 'stream=width,height', '-of', 'csv=p=0', filename]
-    pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=-1)
-    for line in pipe.stdout:
-        w, h = line.decode().strip().split(',')
-        return int(w), int(h)
-
-def read_video(filename):
-    w, h = get_resolution(filename)
-
-    command = ['ffmpeg',
-            '-i', filename,
-            '-f', 'image2pipe',
-            '-pix_fmt', 'bgr24',
-            '-vsync', '0',
-            '-vcodec', 'rawvideo', '-']
-
-    pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=-1)
-    while True:
-        data = pipe.stdout.read(w*h*3)
-        if not data:
-            break
-        yield np.frombuffer(data, dtype='uint8').reshape((h, w, 3))
-
-
-def main(args):
-
-    logger = logging.getLogger(__name__)
-    merge_cfg_from_file(args.cfg)
-    cfg.NUM_GPUS = 1
-    args.weights = cache_url(args.weights, cfg.DOWNLOAD_CACHE)
-    assert_and_infer_cfg(cache_urls=False)
-    model = infer_engine.initialize_model_from_cfg(args.weights)
-    dummy_coco_dataset = dummy_datasets.get_coco_dataset()
-
-
-
-    if os.path.isdir(args.im_or_folder):
-        im_list = glob.iglob(args.im_or_folder + '/*.' + args.image_ext)
-    else:
-        im_list = [args.im_or_folder]
-
-    for video_name in im_list:
-        out_name = os.path.join(
-                args.output_dir, os.path.basename(video_name)
-            )
-        print('Processing {}'.format(video_name))
-
-        boxes = []
-        segments = []
-        keypoints = []
-
-        for frame_i, im in enumerate(read_video(video_name)):
-
-            logger.info('Frame {}'.format(frame_i))
-            timers = defaultdict(Timer)
-            t = time.time()
-            with c2_utils.NamedCudaScope(0):
-                cls_boxes, cls_segms, cls_keyps = infer_engine.im_detect_all(
-                    model, im, None, timers=timers
-                )
-            logger.info('Inference time: {:.3f}s'.format(time.time() - t))
-            for k, v in timers.items():
-                logger.info(' | {}: {:.3f}s'.format(k, v.average_time))
-
-            boxes.append(cls_boxes)
-            segments.append(cls_segms)
-            keypoints.append(cls_keyps)
-
-        
-        # Video resolution
-        metadata = {
-            'w': im.shape[1],
-            'h': im.shape[0],
-        }
-        
-        np.savez_compressed(out_name, boxes=boxes, segments=segments, keypoints=keypoints, metadata=metadata)
-
-
-if __name__ == '__main__':
-    workspace.GlobalInit(['caffe2', '--caffe2_log_level=0'])
-    setup_logging(__name__)
-    args = parse_args()
-    main(args)

VideoPose3D/inference/infer_video_d2.py

@@ -1,152 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-"""Perform inference on a single video or all videos with a certain extension
-(e.g., .mp4) in a folder.
-"""
-
-import detectron2
-from detectron2.utils.logger import setup_logger
-from detectron2.config import get_cfg
-from detectron2 import model_zoo
-from detectron2.engine import DefaultPredictor
-
-import subprocess as sp
-import numpy as np
-import time
-import argparse
-import sys
-import os
-import glob
-
-def parse_args():
-    parser = argparse.ArgumentParser(description='End-to-end inference')
-    parser.add_argument(
-        '--cfg',
-        dest='cfg',
-        help='cfg model file (/path/to/model_config.yaml)',
-        default=None,
-        type=str
-    )
-    parser.add_argument(
-        '--output-dir',
-        dest='output_dir',
-        help='directory for visualization pdfs (default: /tmp/infer_simple)',
-        default='/tmp/infer_simple',
-        type=str
-    )
-    parser.add_argument(
-        '--image-ext',
-        dest='image_ext',
-        help='image file name extension (default: mp4)',
-        default='mp4',
-        type=str
-    )
-    parser.add_argument(
-        'im_or_folder', help='image or folder of images', default=None
-    )
-    if len(sys.argv) == 1:
-        parser.print_help()
-        sys.exit(1)
-    return parser.parse_args()
-
-def get_resolution(filename):
-    command = ['ffprobe', '-v', 'error', '-select_streams', 'v:0',
-               '-show_entries', 'stream=width,height', '-of', 'csv=p=0', filename]
-    pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=-1)
-    for line in pipe.stdout:
-        w, h = line.decode().strip().split(',')
-        return int(w), int(h)
-
-def read_video(filename):
-    w, h = get_resolution(filename)
-
-    command = ['ffmpeg',
-            '-i', filename,
-            '-f', 'image2pipe',
-            '-pix_fmt', 'bgr24',
-            '-vsync', '0',
-            '-vcodec', 'rawvideo', '-']
-
-    pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=-1)
-    while True:
-        data = pipe.stdout.read(w*h*3)
-        if not data:
-            break
-        yield np.frombuffer(data, dtype='uint8').reshape((h, w, 3))
-
-
-def main(args):
-
-    cfg = get_cfg()
-    cfg.merge_from_file(model_zoo.get_config_file(args.cfg))
-    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7
-    cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(args.cfg)
-    predictor = DefaultPredictor(cfg)
-    
-
-    if os.path.isdir(args.im_or_folder):
-        im_list = glob.iglob(args.im_or_folder + '/*.' + args.image_ext)
-    else:
-        im_list = [args.im_or_folder]
-
-    for video_name in im_list:
-        out_name = os.path.join(
-                args.output_dir, os.path.basename(video_name)
-            )
-        print('Processing {}'.format(video_name))
-
-        boxes = []
-        segments = []
-        keypoints = []
-
-        for frame_i, im in enumerate(read_video(video_name)):
-            t = time.time()
-            outputs = predictor(im)['instances'].to('cpu')
-            
-            print('Frame {} processed in {:.3f}s'.format(frame_i, time.time() - t))
-
-            has_bbox = False
-            if outputs.has('pred_boxes'):
-                bbox_tensor = outputs.pred_boxes.tensor.numpy()
-                if len(bbox_tensor) > 0:
-                    has_bbox = True
-                    scores = outputs.scores.numpy()[:, None]
-                    bbox_tensor = np.concatenate((bbox_tensor, scores), axis=1)
-            if has_bbox:
-                kps = outputs.pred_keypoints.numpy()
-                kps_xy = kps[:, :, :2]
-                kps_prob = kps[:, :, 2:3]
-                kps_logit = np.zeros_like(kps_prob) # Dummy
-                kps = np.concatenate((kps_xy, kps_logit, kps_prob), axis=2)
-                kps = kps.transpose(0, 2, 1)
-            else:
-                kps = []
-                bbox_tensor = []
-                
-            # Mimic Detectron1 format
-            cls_boxes = [[], bbox_tensor]
-            cls_keyps = [[], kps]
-            
-            boxes.append(cls_boxes)
-            segments.append(None)
-            keypoints.append(cls_keyps)
-
-        
-        # Video resolution
-        metadata = {
-            'w': im.shape[1],
-            'h': im.shape[0],
-        }
-        
-        np.savez_compressed(out_name, boxes=boxes, segments=segments, keypoints=keypoints, metadata=metadata)
-
-
-if __name__ == '__main__':
-    setup_logger()
-    args = parse_args()
-    main(args)

VideoPose3D/run.py

@@ -1,862 +0,0 @@
-# Copyright (c) 2018-present, Facebook, Inc.
-# All rights reserved.
-#
-# This source code is licensed under the license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-import numpy as np
-
-from common.arguments import parse_args
-import torch
-
-import torch.nn as nn
-import torch.nn.functional as F
-import torch.optim as optim
-import os
-import sys
-import errno
-
-from common.camera import *
-from common.model import *
-from common.loss import *
-from common.generators import ChunkedGenerator, UnchunkedGenerator
-from time import time
-from common.utils import deterministic_random
-
-args = parse_args()
-print(args)
-
-try:
-    # Create checkpoint directory if it does not exist
-    os.makedirs(args.checkpoint)
-except OSError as e:
-    if e.errno != errno.EEXIST:
-        raise RuntimeError('Unable to create checkpoint directory:', args.checkpoint)
-
-print('Loading dataset...')
-dataset_path = 'data/data_3d_' + args.dataset + '.npz'
-if args.dataset == 'h36m':
-    from common.h36m_dataset import Human36mDataset
-    dataset = Human36mDataset(dataset_path)
-elif args.dataset.startswith('humaneva'):
-    from common.humaneva_dataset import HumanEvaDataset
-    dataset = HumanEvaDataset(dataset_path)
-elif args.dataset.startswith('custom'):
-    from common.custom_dataset import CustomDataset
-    dataset = CustomDataset('data/data_2d_' + args.dataset + '_' + args.keypoints + '.npz')
-else:
-    raise KeyError('Invalid dataset')
-
-print('Preparing data...')
-for subject in dataset.subjects():
-    for action in dataset[subject].keys():
-        anim = dataset[subject][action]
-        
-        if 'positions' in anim:
-            positions_3d = []
-            for cam in anim['cameras']:
-                pos_3d = world_to_camera(anim['positions'], R=cam['orientation'], t=cam['translation'])
-                pos_3d[:, 1:] -= pos_3d[:, :1] # Remove global offset, but keep trajectory in first position
-                positions_3d.append(pos_3d)
-            anim['positions_3d'] = positions_3d
-
-print('Loading 2D detections...')
-keypoints = np.load('data/data_2d_' + args.dataset + '_' + args.keypoints + '.npz', allow_pickle=True)
-keypoints_metadata = keypoints['metadata'].item()
-keypoints_symmetry = keypoints_metadata['keypoints_symmetry']
-kps_left, kps_right = list(keypoints_symmetry[0]), list(keypoints_symmetry[1])
-joints_left, joints_right = list(dataset.skeleton().joints_left()), list(dataset.skeleton().joints_right())
-keypoints = keypoints['positions_2d'].item()
-
-for subject in dataset.subjects():
-    assert subject in keypoints, 'Subject {} is missing from the 2D detections dataset'.format(subject)
-    for action in dataset[subject].keys():
-        assert action in keypoints[subject], 'Action {} of subject {} is missing from the 2D detections dataset'.format(action, subject)
-        if 'positions_3d' not in dataset[subject][action]:
-            continue
-            
-        for cam_idx in range(len(keypoints[subject][action])):
-            
-            # We check for >= instead of == because some videos in H3.6M contain extra frames
-            mocap_length = dataset[subject][action]['positions_3d'][cam_idx].shape[0]
-            assert keypoints[subject][action][cam_idx].shape[0] >= mocap_length
-            
-            if keypoints[subject][action][cam_idx].shape[0] > mocap_length:
-                # Shorten sequence
-                keypoints[subject][action][cam_idx] = keypoints[subject][action][cam_idx][:mocap_length]
-
-        assert len(keypoints[subject][action]) == len(dataset[subject][action]['positions_3d'])
-        
-for subject in keypoints.keys():
-    for action in keypoints[subject]:
-        for cam_idx, kps in enumerate(keypoints[subject][action]):
-            # Normalize camera frame
-            cam = dataset.cameras()[subject][cam_idx]
-            kps[..., :2] = normalize_screen_coordinates(kps[..., :2], w=cam['res_w'], h=cam['res_h'])
-            keypoints[subject][action][cam_idx] = kps
-
-subjects_train = args.subjects_train.split(',')
-subjects_semi = [] if not args.subjects_unlabeled else args.subjects_unlabeled.split(',')
-if not args.render:
-    subjects_test = args.subjects_test.split(',')
-else:
-    subjects_test = [args.viz_subject]
-
-semi_supervised = len(subjects_semi) > 0
-if semi_supervised and not dataset.supports_semi_supervised():
-    raise RuntimeError('Semi-supervised training is not implemented for this dataset')
-            
-def fetch(subjects, action_filter=None, subset=1, parse_3d_poses=True):
-    out_poses_3d = []
-    out_poses_2d = []
-    out_camera_params = []
-    for subject in subjects:
-        for action in keypoints[subject].keys():
-            if action_filter is not None:
-                found = False
-                for a in action_filter:
-                    if action.startswith(a):
-                        found = True
-                        break
-                if not found:
-                    continue
-                
-            poses_2d = keypoints[subject][action]
-            for i in range(len(poses_2d)): # Iterate across cameras
-                out_poses_2d.append(poses_2d[i])
-                
-            if subject in dataset.cameras():
-                cams = dataset.cameras()[subject]
-                assert len(cams) == len(poses_2d), 'Camera count mismatch'
-                for cam in cams:
-                    if 'intrinsic' in cam:
-                        out_camera_params.append(cam['intrinsic'])
-                
-            if parse_3d_poses and 'positions_3d' in dataset[subject][action]:
-                poses_3d = dataset[subject][action]['positions_3d']
-                assert len(poses_3d) == len(poses_2d), 'Camera count mismatch'
-                for i in range(len(poses_3d)): # Iterate across cameras
-                    out_poses_3d.append(poses_3d[i])
-    
-    if len(out_camera_params) == 0:
-        out_camera_params = None
-    if len(out_poses_3d) == 0:
-        out_poses_3d = None
-    
-    stride = args.downsample
-    if subset < 1:
-        for i in range(len(out_poses_2d)):
-            n_frames = int(round(len(out_poses_2d[i])//stride * subset)*stride)
-            start = deterministic_random(0, len(out_poses_2d[i]) - n_frames + 1, str(len(out_poses_2d[i])))
-            out_poses_2d[i] = out_poses_2d[i][start:start+n_frames:stride]
-            if out_poses_3d is not None:
-                out_poses_3d[i] = out_poses_3d[i][start:start+n_frames:stride]
-    elif stride > 1:
-        # Downsample as requested
-        for i in range(len(out_poses_2d)):
-            out_poses_2d[i] = out_poses_2d[i][::stride]
-            if out_poses_3d is not None:
-                out_poses_3d[i] = out_poses_3d[i][::stride]
-    
-
-    return out_camera_params, out_poses_3d, out_poses_2d
-
-action_filter = None if args.actions == '*' else args.actions.split(',')
-if action_filter is not None:
-    print('Selected actions:', action_filter)
-    
-cameras_valid, poses_valid, poses_valid_2d = fetch(subjects_test, action_filter)
-
-filter_widths = [int(x) for x in args.architecture.split(',')]
-if not args.disable_optimizations and not args.dense and args.stride == 1:
-    # Use optimized model for single-frame predictions
-    model_pos_train = TemporalModelOptimized1f(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], dataset.skeleton().num_joints(),
-                                filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels)
-else:
-    # When incompatible settings are detected (stride > 1, dense filters, or disabled optimization) fall back to normal model
-    model_pos_train = TemporalModel(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], dataset.skeleton().num_joints(),
-                                filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels,
-                                dense=args.dense)
-    
-model_pos = TemporalModel(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], dataset.skeleton().num_joints(),
-                            filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels,
-                            dense=args.dense)
-
-receptive_field = model_pos.receptive_field()
-print('INFO: Receptive field: {} frames'.format(receptive_field))
-pad = (receptive_field - 1) // 2 # Padding on each side
-if args.causal:
-    print('INFO: Using causal convolutions')
-    causal_shift = pad
-else:
-    causal_shift = 0
-
-model_params = 0
-for parameter in model_pos.parameters():
-    model_params += parameter.numel()
-print('INFO: Trainable parameter count:', model_params)
-
-if torch.cuda.is_available():
-    model_pos = model_pos.cuda()
-    model_pos_train = model_pos_train.cuda()
-    
-if args.resume or args.evaluate:
-    chk_filename = os.path.join(args.checkpoint, args.resume if args.resume else args.evaluate)
-    print('Loading checkpoint', chk_filename)
-    checkpoint = torch.load(chk_filename, map_location=lambda storage, loc: storage)
-    print('This model was trained for {} epochs'.format(checkpoint['epoch']))
-    model_pos_train.load_state_dict(checkpoint['model_pos'])
-    model_pos.load_state_dict(checkpoint['model_pos'])
-    
-    if args.evaluate and 'model_traj' in checkpoint:
-        # Load trajectory model if it contained in the checkpoint (e.g. for inference in the wild)
-        model_traj = TemporalModel(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], 1,
-                            filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels,
-                            dense=args.dense)
-        if torch.cuda.is_available():
-            model_traj = model_traj.cuda()
-        model_traj.load_state_dict(checkpoint['model_traj'])
-    else:
-        model_traj = None
-        
-    
-test_generator = UnchunkedGenerator(cameras_valid, poses_valid, poses_valid_2d,
-                                    pad=pad, causal_shift=causal_shift, augment=False,
-                                    kps_left=kps_left, kps_right=kps_right, joints_left=joints_left, joints_right=joints_right)
-print('INFO: Testing on {} frames'.format(test_generator.num_frames()))
-
-if not args.evaluate:
-    cameras_train, poses_train, poses_train_2d = fetch(subjects_train, action_filter, subset=args.subset)
-
-    lr = args.learning_rate
-    if semi_supervised:
-        cameras_semi, _, poses_semi_2d = fetch(subjects_semi, action_filter, parse_3d_poses=False)
-        
-        if not args.disable_optimizations and not args.dense and args.stride == 1:
-            # Use optimized model for single-frame predictions
-            model_traj_train = TemporalModelOptimized1f(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], 1,
-                    filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels)
-        else:
-            # When incompatible settings are detected (stride > 1, dense filters, or disabled optimization) fall back to normal model
-            model_traj_train = TemporalModel(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], 1,
-                    filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels,
-                    dense=args.dense)
-        
-        model_traj = TemporalModel(poses_valid_2d[0].shape[-2], poses_valid_2d[0].shape[-1], 1,
-                            filter_widths=filter_widths, causal=args.causal, dropout=args.dropout, channels=args.channels,
-                            dense=args.dense)
-        if torch.cuda.is_available():
-            model_traj = model_traj.cuda()
-            model_traj_train = model_traj_train.cuda()
-        optimizer = optim.Adam(list(model_pos_train.parameters()) + list(model_traj_train.parameters()),
-                               lr=lr, amsgrad=True)
-        
-        losses_2d_train_unlabeled = []
-        losses_2d_train_labeled_eval = []
-        losses_2d_train_unlabeled_eval = []
-        losses_2d_valid = []
-
-        losses_traj_train = []
-        losses_traj_train_eval = []
-        losses_traj_valid = []
-    else:
-        optimizer = optim.Adam(model_pos_train.parameters(), lr=lr, amsgrad=True)
-        
-    lr_decay = args.lr_decay
-
-    losses_3d_train = []
-    losses_3d_train_eval = []
-    losses_3d_valid = []
-
-    epoch = 0
-    initial_momentum = 0.1
-    final_momentum = 0.001
-    
-    
-    train_generator = ChunkedGenerator(args.batch_size//args.stride, cameras_train, poses_train, poses_train_2d, args.stride,
-                                       pad=pad, causal_shift=causal_shift, shuffle=True, augment=args.data_augmentation,
-                                       kps_left=kps_left, kps_right=kps_right, joints_left=joints_left, joints_right=joints_right)
-    train_generator_eval = UnchunkedGenerator(cameras_train, poses_train, poses_train_2d,
-                                              pad=pad, causal_shift=causal_shift, augment=False)
-    print('INFO: Training on {} frames'.format(train_generator_eval.num_frames()))
-    if semi_supervised:
-        semi_generator = ChunkedGenerator(args.batch_size//args.stride, cameras_semi, None, poses_semi_2d, args.stride,
-                                          pad=pad, causal_shift=causal_shift, shuffle=True,
-                                          random_seed=4321, augment=args.data_augmentation,
-                                          kps_left=kps_left, kps_right=kps_right, joints_left=joints_left, joints_right=joints_right,
-                                          endless=True)
-        semi_generator_eval = UnchunkedGenerator(cameras_semi, None, poses_semi_2d,
-                                                 pad=pad, causal_shift=causal_shift, augment=False)
-        print('INFO: Semi-supervision on {} frames'.format(semi_generator_eval.num_frames()))
-
-    if args.resume:
-        epoch = checkpoint['epoch']
-        if 'optimizer' in checkpoint and checkpoint['optimizer'] is not None:
-            optimizer.load_state_dict(checkpoint['optimizer'])
-            train_generator.set_random_state(checkpoint['random_state'])
-        else:
-            print('WARNING: this checkpoint does not contain an optimizer state. The optimizer will be reinitialized.')
-        
-        lr = checkpoint['lr']
-        if semi_supervised:
-            model_traj_train.load_state_dict(checkpoint['model_traj'])
-            model_traj.load_state_dict(checkpoint['model_traj'])
-            semi_generator.set_random_state(checkpoint['random_state_semi'])
-            
-    print('** Note: reported losses are averaged over all frames and test-time augmentation is not used here.')
-    print('** The final evaluation will be carried out after the last training epoch.')
-    
-    # Pos model only
-    while epoch < args.epochs:
-        start_time = time()
-        epoch_loss_3d_train = 0
-        epoch_loss_traj_train = 0
-        epoch_loss_2d_train_unlabeled = 0
-        N = 0
-        N_semi = 0
-        model_pos_train.train()
-        if semi_supervised:
-            # Semi-supervised scenario
-            model_traj_train.train()
-            for (_, batch_3d, batch_2d), (cam_semi, _, batch_2d_semi) in \
-                zip(train_generator.next_epoch(), semi_generator.next_epoch()):
-                
-                # Fall back to supervised training for the first epoch (to avoid instability)
-                skip = epoch < args.warmup
-                
-                cam_semi = torch.from_numpy(cam_semi.astype('float32'))
-                inputs_3d = torch.from_numpy(batch_3d.astype('float32'))
-                if torch.cuda.is_available():
-                    cam_semi = cam_semi.cuda()
-                    inputs_3d = inputs_3d.cuda()
-                    
-                inputs_traj = inputs_3d[:, :, :1].clone()
-                inputs_3d[:, :, 0] = 0
-                
-                # Split point between labeled and unlabeled samples in the batch
-                split_idx = inputs_3d.shape[0]
-
-                inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
-                inputs_2d_semi = torch.from_numpy(batch_2d_semi.astype('float32'))
-                if torch.cuda.is_available():
-                    inputs_2d = inputs_2d.cuda()
-                    inputs_2d_semi = inputs_2d_semi.cuda()
-                inputs_2d_cat =  torch.cat((inputs_2d, inputs_2d_semi), dim=0) if not skip else inputs_2d
-
-                optimizer.zero_grad()
-
-                # Compute 3D poses
-                predicted_3d_pos_cat = model_pos_train(inputs_2d_cat)
-
-                loss_3d_pos = mpjpe(predicted_3d_pos_cat[:split_idx], inputs_3d)
-                epoch_loss_3d_train += inputs_3d.shape[0]*inputs_3d.shape[1] * loss_3d_pos.item()
-                N += inputs_3d.shape[0]*inputs_3d.shape[1]
-                loss_total = loss_3d_pos
-
-                # Compute global trajectory
-                predicted_traj_cat = model_traj_train(inputs_2d_cat)
-                w = 1 / inputs_traj[:, :, :, 2] # Weight inversely proportional to depth
-                loss_traj = weighted_mpjpe(predicted_traj_cat[:split_idx], inputs_traj, w)
-                epoch_loss_traj_train += inputs_3d.shape[0]*inputs_3d.shape[1] * loss_traj.item()
-                assert inputs_traj.shape[0]*inputs_traj.shape[1] == inputs_3d.shape[0]*inputs_3d.shape[1]
-                loss_total += loss_traj
-
-                if not skip:
-                    # Semi-supervised loss for unlabeled samples
-                    predicted_semi = predicted_3d_pos_cat[split_idx:]
-                    if pad > 0:
-                        target_semi = inputs_2d_semi[:, pad:-pad, :, :2].contiguous()
-                    else:
-                        target_semi = inputs_2d_semi[:, :, :, :2].contiguous()
-                        
-                    projection_func = project_to_2d_linear if args.linear_projection else project_to_2d
-                    reconstruction_semi = projection_func(predicted_semi + predicted_traj_cat[split_idx:], cam_semi)
-
-                    loss_reconstruction = mpjpe(reconstruction_semi, target_semi) # On 2D poses
-                    epoch_loss_2d_train_unlabeled += predicted_semi.shape[0]*predicted_semi.shape[1] * loss_reconstruction.item()
-                    if not args.no_proj:
-                        loss_total += loss_reconstruction
-                    
-                    # Bone length term to enforce kinematic constraints
-                    if args.bone_length_term:
-                        dists = predicted_3d_pos_cat[:, :, 1:] - predicted_3d_pos_cat[:, :, dataset.skeleton().parents()[1:]]
-                        bone_lengths = torch.mean(torch.norm(dists, dim=3), dim=1)
-                        penalty = torch.mean(torch.abs(torch.mean(bone_lengths[:split_idx], dim=0) \
-                                                     - torch.mean(bone_lengths[split_idx:], dim=0)))
-                        loss_total += penalty
-                        
-                    
-                    N_semi += predicted_semi.shape[0]*predicted_semi.shape[1]
-                else:
-                    N_semi += 1 # To avoid division by zero
-
-                loss_total.backward()
-
-                optimizer.step()
-            losses_traj_train.append(epoch_loss_traj_train / N)
-            losses_2d_train_unlabeled.append(epoch_loss_2d_train_unlabeled / N_semi)
-        else:
-            # Regular supervised scenario
-            for _, batch_3d, batch_2d in train_generator.next_epoch():
-                inputs_3d = torch.from_numpy(batch_3d.astype('float32'))
-                inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
-                if torch.cuda.is_available():
-                    inputs_3d = inputs_3d.cuda()
-                    inputs_2d = inputs_2d.cuda()
-                inputs_3d[:, :, 0] = 0
-
-                optimizer.zero_grad()
-
-                # Predict 3D poses
-                predicted_3d_pos = model_pos_train(inputs_2d)
-                loss_3d_pos = mpjpe(predicted_3d_pos, inputs_3d)
-                epoch_loss_3d_train += inputs_3d.shape[0]*inputs_3d.shape[1] * loss_3d_pos.item()
-                N += inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                loss_total = loss_3d_pos
-                loss_total.backward()
-
-                optimizer.step()
-
-        losses_3d_train.append(epoch_loss_3d_train / N)
-
-        # End-of-epoch evaluation
-        with torch.no_grad():
-            model_pos.load_state_dict(model_pos_train.state_dict())
-            model_pos.eval()
-            if semi_supervised:
-                model_traj.load_state_dict(model_traj_train.state_dict())
-                model_traj.eval()
-
-            epoch_loss_3d_valid = 0
-            epoch_loss_traj_valid = 0
-            epoch_loss_2d_valid = 0
-            N = 0
-            
-            if not args.no_eval:
-                # Evaluate on test set
-                for cam, batch, batch_2d in test_generator.next_epoch():
-                    inputs_3d = torch.from_numpy(batch.astype('float32'))
-                    inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
-                    if torch.cuda.is_available():
-                        inputs_3d = inputs_3d.cuda()
-                        inputs_2d = inputs_2d.cuda()
-                    inputs_traj = inputs_3d[:, :, :1].clone()
-                    inputs_3d[:, :, 0] = 0
-
-                    # Predict 3D poses
-                    predicted_3d_pos = model_pos(inputs_2d)
-                    loss_3d_pos = mpjpe(predicted_3d_pos, inputs_3d)
-                    epoch_loss_3d_valid += inputs_3d.shape[0]*inputs_3d.shape[1] * loss_3d_pos.item()
-                    N += inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                    if semi_supervised:
-                        cam = torch.from_numpy(cam.astype('float32'))
-                        if torch.cuda.is_available():
-                            cam = cam.cuda()
-
-                        predicted_traj = model_traj(inputs_2d)
-                        loss_traj = mpjpe(predicted_traj, inputs_traj)
-                        epoch_loss_traj_valid += inputs_traj.shape[0]*inputs_traj.shape[1] * loss_traj.item()
-                        assert inputs_traj.shape[0]*inputs_traj.shape[1] == inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                        if pad > 0:
-                            target = inputs_2d[:, pad:-pad, :, :2].contiguous()
-                        else:
-                            target = inputs_2d[:, :, :, :2].contiguous()
-                        reconstruction = project_to_2d(predicted_3d_pos + predicted_traj, cam)
-                        loss_reconstruction = mpjpe(reconstruction, target) # On 2D poses
-                        epoch_loss_2d_valid += reconstruction.shape[0]*reconstruction.shape[1] * loss_reconstruction.item()
-                        assert reconstruction.shape[0]*reconstruction.shape[1] == inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                losses_3d_valid.append(epoch_loss_3d_valid / N)
-                if semi_supervised:
-                    losses_traj_valid.append(epoch_loss_traj_valid / N)
-                    losses_2d_valid.append(epoch_loss_2d_valid / N)
-
-
-                # Evaluate on training set, this time in evaluation mode
-                epoch_loss_3d_train_eval = 0
-                epoch_loss_traj_train_eval = 0
-                epoch_loss_2d_train_labeled_eval = 0
-                N = 0
-                for cam, batch, batch_2d in train_generator_eval.next_epoch():
-                    if batch_2d.shape[1] == 0:
-                        # This can only happen when downsampling the dataset
-                        continue
-                        
-                    inputs_3d = torch.from_numpy(batch.astype('float32'))
-                    inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
-                    if torch.cuda.is_available():
-                        inputs_3d = inputs_3d.cuda()
-                        inputs_2d = inputs_2d.cuda()
-                    inputs_traj = inputs_3d[:, :, :1].clone()
-                    inputs_3d[:, :, 0] = 0
-
-                    # Compute 3D poses
-                    predicted_3d_pos = model_pos(inputs_2d)
-                    loss_3d_pos = mpjpe(predicted_3d_pos, inputs_3d)
-                    epoch_loss_3d_train_eval += inputs_3d.shape[0]*inputs_3d.shape[1] * loss_3d_pos.item()
-                    N += inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                    if semi_supervised:
-                        cam = torch.from_numpy(cam.astype('float32'))
-                        if torch.cuda.is_available():
-                            cam = cam.cuda()
-                        predicted_traj = model_traj(inputs_2d)
-                        loss_traj = mpjpe(predicted_traj, inputs_traj)
-                        epoch_loss_traj_train_eval += inputs_traj.shape[0]*inputs_traj.shape[1] * loss_traj.item()
-                        assert inputs_traj.shape[0]*inputs_traj.shape[1] == inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                        if pad > 0:
-                            target = inputs_2d[:, pad:-pad, :, :2].contiguous()
-                        else:
-                            target = inputs_2d[:, :, :, :2].contiguous()
-                        reconstruction = project_to_2d(predicted_3d_pos + predicted_traj, cam)
-                        loss_reconstruction = mpjpe(reconstruction, target)
-                        epoch_loss_2d_train_labeled_eval += reconstruction.shape[0]*reconstruction.shape[1] * loss_reconstruction.item()
-                        assert reconstruction.shape[0]*reconstruction.shape[1] == inputs_3d.shape[0]*inputs_3d.shape[1]
-
-                losses_3d_train_eval.append(epoch_loss_3d_train_eval / N)
-                if semi_supervised:
-                    losses_traj_train_eval.append(epoch_loss_traj_train_eval / N)
-                    losses_2d_train_labeled_eval.append(epoch_loss_2d_train_labeled_eval / N)
-
-                # Evaluate 2D loss on unlabeled training set (in evaluation mode)
-                epoch_loss_2d_train_unlabeled_eval = 0
-                N_semi = 0
-                if semi_supervised:
-                    for cam, _, batch_2d in semi_generator_eval.next_epoch():
-                        cam = torch.from_numpy(cam.astype('float32'))
-                        inputs_2d_semi = torch.from_numpy(batch_2d.astype('float32'))
-                        if torch.cuda.is_available():
-                            cam = cam.cuda()
-                            inputs_2d_semi = inputs_2d_semi.cuda()
-
-                        predicted_3d_pos_semi = model_pos(inputs_2d_semi)
-                        predicted_traj_semi = model_traj(inputs_2d_semi)
-                        if pad > 0:
-                            target_semi = inputs_2d_semi[:, pad:-pad, :, :2].contiguous()
-                        else:
-                            target_semi = inputs_2d_semi[:, :, :, :2].contiguous()
-                        reconstruction_semi = project_to_2d(predicted_3d_pos_semi + predicted_traj_semi, cam)
-                        loss_reconstruction_semi = mpjpe(reconstruction_semi, target_semi)
-
-                        epoch_loss_2d_train_unlabeled_eval += reconstruction_semi.shape[0]*reconstruction_semi.shape[1] \
-                                                              * loss_reconstruction_semi.item()
-                        N_semi += reconstruction_semi.shape[0]*reconstruction_semi.shape[1]
-                    losses_2d_train_unlabeled_eval.append(epoch_loss_2d_train_unlabeled_eval / N_semi)
-
-        elapsed = (time() - start_time)/60
-        
-        if args.no_eval:
-            print('[%d] time %.2f lr %f 3d_train %f' % (
-                    epoch + 1,
-                    elapsed,
-                    lr,
-                    losses_3d_train[-1] * 1000))
-        else:
-            if semi_supervised:
-                print('[%d] time %.2f lr %f 3d_train %f 3d_eval %f traj_eval %f 3d_valid %f '
-                      'traj_valid %f 2d_train_sup %f 2d_train_unsup %f 2d_valid %f' % (
-                        epoch + 1,
-                        elapsed,
-                        lr,
-                        losses_3d_train[-1] * 1000,
-                        losses_3d_train_eval[-1] * 1000,
-                        losses_traj_train_eval[-1] * 1000,
-                        losses_3d_valid[-1] * 1000,
-                        losses_traj_valid[-1] * 1000,
-                        losses_2d_train_labeled_eval[-1],
-                        losses_2d_train_unlabeled_eval[-1],
-                        losses_2d_valid[-1]))
-            else:
-                print('[%d] time %.2f lr %f 3d_train %f 3d_eval %f 3d_valid %f' % (
-                        epoch + 1,
-                        elapsed,
-                        lr,
-                        losses_3d_train[-1] * 1000,
-                        losses_3d_train_eval[-1] * 1000,
-                        losses_3d_valid[-1]  *1000))
-        
-        # Decay learning rate exponentially
-        lr *= lr_decay
-        for param_group in optimizer.param_groups:
-            param_group['lr'] *= lr_decay
-        epoch += 1
-        
-        # Decay BatchNorm momentum
-        momentum = initial_momentum * np.exp(-epoch/args.epochs * np.log(initial_momentum/final_momentum))
-        model_pos_train.set_bn_momentum(momentum)
-        if semi_supervised:
-            model_traj_train.set_bn_momentum(momentum)
-            
-        # Save checkpoint if necessary
-        if epoch % args.checkpoint_frequency == 0:
-            chk_path = os.path.join(args.checkpoint, 'epoch_{}.bin'.format(epoch))
-            print('Saving checkpoint to', chk_path)
-            
-            torch.save({
-                'epoch': epoch,
-                'lr': lr,
-                'random_state': train_generator.random_state(),
-                'optimizer': optimizer.state_dict(),
-                'model_pos': model_pos_train.state_dict(),
-                'model_traj': model_traj_train.state_dict() if semi_supervised else None,
-                'random_state_semi': semi_generator.random_state() if semi_supervised else None,
-            }, chk_path)
-            
-        # Save training curves after every epoch, as .png images (if requested)
-        if args.export_training_curves and epoch > 3:
-            if 'matplotlib' not in sys.modules:
-                import matplotlib
-                matplotlib.use('Agg')
-                import matplotlib.pyplot as plt
-            
-            plt.figure()
-            epoch_x = np.arange(3, len(losses_3d_train)) + 1
-            plt.plot(epoch_x, losses_3d_train[3:], '--', color='C0')
-            plt.plot(epoch_x, losses_3d_train_eval[3:], color='C0')
-            plt.plot(epoch_x, losses_3d_valid[3:], color='C1')
-            plt.legend(['3d train', '3d train (eval)', '3d valid (eval)'])
-            plt.ylabel('MPJPE (m)')
-            plt.xlabel('Epoch')
-            plt.xlim((3, epoch))
-            plt.savefig(os.path.join(args.checkpoint, 'loss_3d.png'))
-
-            if semi_supervised:
-                plt.figure()
-                plt.plot(epoch_x, losses_traj_train[3:], '--', color='C0')
-                plt.plot(epoch_x, losses_traj_train_eval[3:], color='C0')
-                plt.plot(epoch_x, losses_traj_valid[3:], color='C1')
-                plt.legend(['traj. train', 'traj. train (eval)', 'traj. valid (eval)'])
-                plt.ylabel('Mean distance (m)')
-                plt.xlabel('Epoch')
-                plt.xlim((3, epoch))
-                plt.savefig(os.path.join(args.checkpoint, 'loss_traj.png'))
-
-                plt.figure()
-                plt.plot(epoch_x, losses_2d_train_labeled_eval[3:], color='C0')
-                plt.plot(epoch_x, losses_2d_train_unlabeled[3:], '--', color='C1')
-                plt.plot(epoch_x, losses_2d_train_unlabeled_eval[3:], color='C1')
-                plt.plot(epoch_x, losses_2d_valid[3:], color='C2')
-                plt.legend(['2d train labeled (eval)', '2d train unlabeled', '2d train unlabeled (eval)', '2d valid (eval)'])
-                plt.ylabel('MPJPE (2D)')
-                plt.xlabel('Epoch')
-                plt.xlim((3, epoch))
-                plt.savefig(os.path.join(args.checkpoint, 'loss_2d.png'))
-            plt.close('all')
-
-# Evaluate
-def evaluate(test_generator, action=None, return_predictions=False, use_trajectory_model=False):
-    epoch_loss_3d_pos = 0
-    epoch_loss_3d_pos_procrustes = 0
-    epoch_loss_3d_pos_scale = 0
-    epoch_loss_3d_vel = 0
-    with torch.no_grad():
-        if not use_trajectory_model:
-            model_pos.eval()
-        else:
-            model_traj.eval()
-        N = 0
-        for _, batch, batch_2d in test_generator.next_epoch():
-            inputs_2d = torch.from_numpy(batch_2d.astype('float32'))
-            if torch.cuda.is_available():
-                inputs_2d = inputs_2d.cuda()
-
-            # Positional model
-            if not use_trajectory_model:
-                predicted_3d_pos = model_pos(inputs_2d)
-            else:
-                predicted_3d_pos = model_traj(inputs_2d)
-
-            # Test-time augmentation (if enabled)
-            if test_generator.augment_enabled():
-                # Undo flipping and take average with non-flipped version
-                predicted_3d_pos[1, :, :, 0] *= -1
-                if not use_trajectory_model:
-                    predicted_3d_pos[1, :, joints_left + joints_right] = predicted_3d_pos[1, :, joints_right + joints_left]
-                predicted_3d_pos = torch.mean(predicted_3d_pos, dim=0, keepdim=True)
-                
-            if return_predictions:
-                return predicted_3d_pos.squeeze(0).cpu().numpy()
-                
-            inputs_3d = torch.from_numpy(batch.astype('float32'))
-            if torch.cuda.is_available():
-                inputs_3d = inputs_3d.cuda()
-            inputs_3d[:, :, 0] = 0    
-            if test_generator.augment_enabled():
-                inputs_3d = inputs_3d[:1]
-
-            error = mpjpe(predicted_3d_pos, inputs_3d)
-            epoch_loss_3d_pos_scale += inputs_3d.shape[0]*inputs_3d.shape[1] * n_mpjpe(predicted_3d_pos, inputs_3d).item()
-
-            epoch_loss_3d_pos += inputs_3d.shape[0]*inputs_3d.shape[1] * error.item()
-            N += inputs_3d.shape[0] * inputs_3d.shape[1]
-            
-            inputs = inputs_3d.cpu().numpy().reshape(-1, inputs_3d.shape[-2], inputs_3d.shape[-1])
-            predicted_3d_pos = predicted_3d_pos.cpu().numpy().reshape(-1, inputs_3d.shape[-2], inputs_3d.shape[-1])
-
-            epoch_loss_3d_pos_procrustes += inputs_3d.shape[0]*inputs_3d.shape[1] * p_mpjpe(predicted_3d_pos, inputs)
-
-            # Compute velocity error
-            epoch_loss_3d_vel += inputs_3d.shape[0]*inputs_3d.shape[1] * mean_velocity_error(predicted_3d_pos, inputs)
-            
-    if action is None:
-        print('----------')
-    else:
-        print('----'+action+'----')
-    e1 = (epoch_loss_3d_pos / N)*1000
-    e2 = (epoch_loss_3d_pos_procrustes / N)*1000
-    e3 = (epoch_loss_3d_pos_scale / N)*1000
-    ev = (epoch_loss_3d_vel / N)*1000
-    print('Test time augmentation:', test_generator.augment_enabled())
-    print('Protocol #1 Error (MPJPE):', e1, 'mm')
-    print('Protocol #2 Error (P-MPJPE):', e2, 'mm')
-    print('Protocol #3 Error (N-MPJPE):', e3, 'mm')
-    print('Velocity Error (MPJVE):', ev, 'mm')
-    print('----------')
-
-    return e1, e2, e3, ev
-
-
-if args.render:
-    print('Rendering...')
-    
-    input_keypoints = keypoints[args.viz_subject][args.viz_action][args.viz_camera].copy()
-    ground_truth = None
-    if args.viz_subject in dataset.subjects() and args.viz_action in dataset[args.viz_subject]:
-        if 'positions_3d' in dataset[args.viz_subject][args.viz_action]:
-            ground_truth = dataset[args.viz_subject][args.viz_action]['positions_3d'][args.viz_camera].copy()
-    if ground_truth is None:
-        print('INFO: this action is unlabeled. Ground truth will not be rendered.')
-        
-    gen = UnchunkedGenerator(None, None, [input_keypoints],
-                             pad=pad, causal_shift=causal_shift, augment=args.test_time_augmentation,
-                             kps_left=kps_left, kps_right=kps_right, joints_left=joints_left, joints_right=joints_right)
-    prediction = evaluate(gen, return_predictions=True)
-    if model_traj is not None and ground_truth is None:
-        prediction_traj = evaluate(gen, return_predictions=True, use_trajectory_model=True)
-        prediction += prediction_traj
-    
-    if args.viz_export is not None:
-        print('Exporting joint positions to', args.viz_export)
-        # Predictions are in camera space
-        np.save(args.viz_export, prediction)
-    
-    if args.viz_output is not None:
-        if ground_truth is not None:
-            # Reapply trajectory
-            trajectory = ground_truth[:, :1]
-            ground_truth[:, 1:] += trajectory
-            prediction += trajectory
-        
-        # Invert camera transformation
-        cam = dataset.cameras()[args.viz_subject][args.viz_camera]
-        if ground_truth is not None:
-            prediction = camera_to_world(prediction, R=cam['orientation'], t=cam['translation'])
-            ground_truth = camera_to_world(ground_truth, R=cam['orientation'], t=cam['translation'])
-        else:
-            # If the ground truth is not available, take the camera extrinsic params from a random subject.
-            # They are almost the same, and anyway, we only need this for visualization purposes.
-            for subject in dataset.cameras():
-                if 'orientation' in dataset.cameras()[subject][args.viz_camera]:
-                    rot = dataset.cameras()[subject][args.viz_camera]['orientation']
-                    break
-            prediction = camera_to_world(prediction, R=rot, t=0)
-            # We don't have the trajectory, but at least we can rebase the height
-            prediction[:, :, 2] -= np.min(prediction[:, :, 2])
-        
-        anim_output = {'Reconstruction': prediction}
-        if ground_truth is not None and not args.viz_no_ground_truth:
-            anim_output['Ground truth'] = ground_truth
-        
-        input_keypoints = image_coordinates(input_keypoints[..., :2], w=cam['res_w'], h=cam['res_h'])
-        
-        from common.visualization import render_animation
-        render_animation(input_keypoints, keypoints_metadata, anim_output,
-                         dataset.skeleton(), dataset.fps(), args.viz_bitrate, cam['azimuth'], args.viz_output,
-                         limit=args.viz_limit, downsample=args.viz_downsample, size=args.viz_size,
-                         input_video_path=args.viz_video, viewport=(cam['res_w'], cam['res_h']),
-                         input_video_skip=args.viz_skip)
-    
-else:
-    print('Evaluating...')
-    all_actions = {}
-    all_actions_by_subject = {}
-    for subject in subjects_test:
-        if subject not in all_actions_by_subject:
-            all_actions_by_subject[subject] = {}
-
-        for action in dataset[subject].keys():
-            action_name = action.split(' ')[0]
-            if action_name not in all_actions:
-                all_actions[action_name] = []
-            if action_name not in all_actions_by_subject[subject]:
-                all_actions_by_subject[subject][action_name] = []
-            all_actions[action_name].append((subject, action))
-            all_actions_by_subject[subject][action_name].append((subject, action))
-
-    def fetch_actions(actions):
-        out_poses_3d = []
-        out_poses_2d = []
-
-        for subject, action in actions:
-            poses_2d = keypoints[subject][action]
-            for i in range(len(poses_2d)): # Iterate across cameras
-                out_poses_2d.append(poses_2d[i])
-
-            poses_3d = dataset[subject][action]['positions_3d']
-            assert len(poses_3d) == len(poses_2d), 'Camera count mismatch'
-            for i in range(len(poses_3d)): # Iterate across cameras
-                out_poses_3d.append(poses_3d[i])
-
-        stride = args.downsample
-        if stride > 1:
-            # Downsample as requested
-            for i in range(len(out_poses_2d)):
-                out_poses_2d[i] = out_poses_2d[i][::stride]
-                if out_poses_3d is not None:
-                    out_poses_3d[i] = out_poses_3d[i][::stride]
-        
-        return out_poses_3d, out_poses_2d
-
-    def run_evaluation(actions, action_filter=None):
-        errors_p1 = []
-        errors_p2 = []
-        errors_p3 = []
-        errors_vel = []
-
-        for action_key in actions.keys():
-            if action_filter is not None:
-                found = False
-                for a in action_filter:
-                    if action_key.startswith(a):
-                        found = True
-                        break
-                if not found:
-                    continue
-
-            poses_act, poses_2d_act = fetch_actions(actions[action_key])
-            gen = UnchunkedGenerator(None, poses_act, poses_2d_act,
-                                     pad=pad, causal_shift=causal_shift, augment=args.test_time_augmentation,
-                                     kps_left=kps_left, kps_right=kps_right, joints_left=joints_left, joints_right=joints_right)
-            e1, e2, e3, ev = evaluate(gen, action_key)
-            errors_p1.append(e1)
-            errors_p2.append(e2)
-            errors_p3.append(e3)
-            errors_vel.append(ev)
-
-        print('Protocol #1   (MPJPE) action-wise average:', round(np.mean(errors_p1), 1), 'mm')
-        print('Protocol #2 (P-MPJPE) action-wise average:', round(np.mean(errors_p2), 1), 'mm')
-        print('Protocol #3 (N-MPJPE) action-wise average:', round(np.mean(errors_p3), 1), 'mm')
-        print('Velocity      (MPJVE) action-wise average:', round(np.mean(errors_vel), 2), 'mm')
-
-    if not args.by_subject:
-        run_evaluation(all_actions, action_filter)
-    else:
-        for subject in all_actions_by_subject.keys():
-            print('Evaluating on subject', subject)
-            run_evaluation(all_actions_by_subject[subject], action_filter)
-            print('')

frontend/package.json

@@ -8,17 +8,21 @@
     "lint": "vue-cli-service lint"
   },
   "dependencies": {
+    "@antv/data-set": "^0.11.8",
+    "@antv/g2": "^4.2.0",
     "ant-design-vue": "^1.7.8",
     "axios": "^0.21.4",
     "babel-plugin-import": "^1.13.3",
     "core-js": "^3.6.5",
     "element-ui": "^2.15.6",
     "vue": "^2.6.11",
-    "vue-axios": "^3.3.6"
+    "vue-axios": "^3.3.6",
+    "vue-router": "^3.2.0"
   },
   "devDependencies": {
     "@vue/cli-plugin-babel": "~4.5.0",
     "@vue/cli-plugin-eslint": "~4.5.0",
+    "@vue/cli-plugin-router": "~4.5.13",
     "@vue/cli-service": "~4.5.0",
     "babel-eslint": "^10.1.0",
     "eslint": "^6.7.2",

frontend/src/App.vue

@@ -1,68 +1,54 @@
 <template>
-  <div id="app">
-    <header id="header">
-      <h1>Pose Correction</h1>
-    </header>
-    <div id="content">
-      <div id="image-upload">
-        <uploadCard @infor="getInfor"/>
-      </div>
-      <div id="info-display">
-        <displayCard ref="dCard"/>
-      </div>
-    </div>
-    <footer id="footer">
-      Nanjing University, Software Institute
-    </footer>
+  <div id="nav">
+    <h2>儿童运动能力评估系统</h2>
+    <router-link class="nav-item" style="margin-right:30px" to="/static">站立姿态评估</router-link>
+    <router-link class="nav-item" to="/dynamic">运动姿态评估</router-link>
+    <router-view/>
   </div>
 </template>
 
-<script>
-import uploadCard from './components/upload/uploadCard.vue'
-import displayCard from './components/display/displayCard.vue'
-
-export default {
-  mounted(){
-  },
-  data() {
-    return {
-      // displayFlag: false
-    }
-  },
-  name: 'App',
-  components: {
-    uploadCard,
-    displayCard
-  },
-  methods:{
-    getInfor(id) {
-      console.log(id)
-      this.$refs.dCard.getInfor(id)
-    }
-  }
-}
-</script>
-
 <style>
-#app {
+/* #app {
   font-family: Avenir, Helvetica, Arial, sans-serif;
   -webkit-font-smoothing: antialiased;
   -moz-osx-font-smoothing: grayscale;
   text-align: center;
   color: #2c3e50;
-  /* margin-top: 60px; */
-}
-#header{
-  margin-top: 2%;
-  height: 50px;
+} */
+
+#nav {
+  margin-top: 20px;
+  /* margin-bottom: 10px; */
+  /* padding-bottom: 10px; */
+  text-align: center;
+  /* height:30px; */
 }
-#content{
-  display: grid;
-  height: 500px;
-  grid-template-columns: 1fr 1fr;
+
+#nav .nav-item {
+  display: inline-block;
+  width: 200px;
+  height: 30px;
+  line-height: 30px;
+  font-weight: bold;
+  color: white;
+  text-decoration: none;
+  background: skyblue;
+  border-radius: 3px;
+
 }
-#footer{
-  margin-top: 20px;
-  padding-top: 30px;
+
+#nav a.router-link-exact-active {
+  background: dodgerblue;
 }
 </style>
+
+<script>
+// @ is an alias to /src
+
+export default {
+  name: 'Home',
+  components: {
+  }
+}
+</script>
+

frontend/src/components/display/displayCard.vue

@@ -1,241 +0,0 @@
-<template>
-  <el-card class="box-card">
-    <div slot="header" class="clearfix">
-      <span>Correction Analysis</span>
-    </div>
-    <el-input
-      id="search_bar"
-      placeholder="Enter your analysis ID"
-      v-model="correction_id"
-      style="width: 250px"
-    >
-      <el-button
-        slot="suffix"
-        icon="el-icon-search"
-        circle
-        @click="getAnalysis"
-        size="small"
-        style="margin-top:4px"
-      ></el-button>
-    </el-input>
-    <div id="corrected-image-container">
-      <div class="corrected-image-group">
-        <el-image
-          :src="images['front-correction']"
-          class="corrected-image"
-          id="front-correction"
-        >
-          <div slot="placeholder" class="image-slot">
-            Image is under analysis
-          </div>
-        </el-image>
-        <div class="corrected-text-group">
-          <el-button
-            size="small"
-            type="primary"
-            icon="el-icon-download"
-            class="correction-download"
-            @click="downloadImage('front-correction')"
-            >Download</el-button
-          >
-        </div>
-      </div>
-      <div class="corrected-image-group" style="margin-left: 3%">
-        <el-image
-          :src="images['right-correction']"
-          class="corrected-image"
-          id="right-correction"
-        >
-          <div slot="placeholder" class="image-slot">
-            Image is under analysis
-          </div>
-        </el-image>
-        <div class="corrected-text-group">
-          <el-button
-            size="small"
-            type="primary"
-            icon="el-icon-download"
-            class="correction-download"
-            @click="downloadImage('right-correction')"
-            >Download</el-button
-          >
-        </div>
-      </div>
-    </div>
-    <div id="information-container">
-      <div id="level-desc">
-        <p class="level-item" :style="{ 'background-color': level_color[0] }">
-          Normal
-        </p>
-        <p class="level-item" :style="{ 'background-color': level_color[1] }">
-          Mild
-        </p>
-        <p class="level-item" :style="{ 'background-color': level_color[2] }">
-          Moderate
-        </p>
-        <p class="level-item" :style="{ 'background-color': level_color[3] }">
-          Severe
-        </p>
-      </div>
-      <div v-for="item in information" :key="item.index" class="text-item">
-        <div class="item-title">{{ item.title }}</div>
-        <div
-          v-for="detail in item.detail"
-          :key="detail.index"
-          class="detail-item"
-          :style="{ 'margin-top': '3%' }"
-        >
-          <div>
-            {{ detail.name }}
-          </div>
-          <div
-            class="detail-value"
-            :style="{
-              'background-color': level_color[detail.level],
-              color: 'white',
-              'border-radius': '2px',
-              padding: '2px 4px 2px 4px',
-            }"
-          >
-            {{ detail.value }}
-          </div>
-        </div>
-      </div>
-    </div>
-  </el-card>
-</template>
-
-<style>
-.text {
-  font-size: 14px;
-}
-
-.item {
-  margin-bottom: 18px;
-}
-
-.clearfix:after {
-  clear: both;
-}
-
-/* .box-card {
-  width: 480px;
-} */
-#search_bar {
-  display: flex;
-  justify-content: center;
-  align-items: center;
-  margin-bottom: 5%;
-}
-.box-card {
-  margin-left: 10%;
-  margin-right: 10%;
-  height: 520px;
-  overflow: auto;
-}
-#level-desc {
-  display: flex;
-  justify-content: baseline;
-  align-items: center;
-}
-.level-item {
-  color: white;
-  margin-left: 3%;
-  border-radius: 3px;
-  font-size: 14px;
-  padding: 2px 4px 2px 4px;
-}
-#corrected-image-container {
-  display: flex;
-  justify-content: center;
-  align-items: center;
-  border: 1px solid lightblue;
-  border-radius: 5px;
-  margin-bottom: 5%;
-}
-.corrected-image-group {
-  width: 45%;
-  text-align: center;
-  padding: 8px 10px 5px 10px;
-}
-.corrected-image {
-  height: 200px;
-  text-align: center;
-  width: 100%;
-}
-.corrected-text-group {
-  text-align: center;
-}
-/* .corrected-text-group .correction-download {
-  float: right;
-} */
-
-#information-container {
-  overflow: auto;
-  border: 1px solid lightblue;
-  border-radius: 5px;
-}
-.text-item {
-  margin-bottom: 10px;
-  padding: 5px 8px 5px 5px;
-}
-.item-title {
-  display: block;
-  margin-bottom: 8px;
-  font-weight: bold;
-}
-
-.detail-item {
-  display: grid;
-  grid-template-columns: 1fr 3fr;
-}
-</style>
-<script>
-import axios from "axios";
-export default {
-  mounted() {},
-  data() {
-    return {
-      information: [],
-      images: {
-        "front-correction": "",
-        "right-correction": "",
-      },
-      level_color: ["#67C23A", "#CDDC39", "#E6A23C", "#F56C6C"],
-      correction_id: "",
-    };
-  },
-  props: {},
-  methods: {
-    getInfor(id){
-      console.log(id)
-      this.correction_id = id;
-    },
-    getAnalysis() {
-      let id = this.correction_id;
-      let _this = this;
-      let formData = new FormData();
-      formData.append("id", id);
-      axios
-        .post("http://106.15.1.178:8090/get_result", formData)
-        .then((response) => {
-          const data = response.data;
-          _this.information = JSON.parse(data.data);
-          _this.images["front-correction"] =
-            "data:image/png;base64," + data.front;
-          _this.images["right-correction"] =
-            "data:image/png;base64," + data.right;
-          console.log(_this.information);
-        });
-    },
-    downloadImage(name) {
-      // let _this  = this;
-      const url = this.images[name];
-      let a = document.createElement("a");
-      a.download = "";
-      a.href = url;
-      a.click();
-    },
-  },
-};
-</script>

frontend/src/components/upload/imageUpload.vue

@@ -1,49 +0,0 @@
-<template>
-  <el-upload
-    class="upload-demo"
-    action="http://106.15.1.178:8090/send_form"
-    :on-change="handleChange"
-    :on-preview="handlePreview"
-    :on-remove="handleRemove"
-    multiple
-    :auto-upload="false"
-    :limit="2"
-    :on-exceed="handleExceed"
-    :file-list="fileList"
-    :data="{filename:filename}"
-    ref="upload"
-  >
-    <el-button size="small" type="primary" icon="el-icon-upload2">Click to upload</el-button>
-  </el-upload>
-</template>
-<script>
-export default {
-  data() {
-    return {
-      fileList: [
-      ],
-      filename: ''
-    };
-  },
-  methods: {
-    handleRemove(file, fileList) {
-      console.log(file, fileList);
-    },
-    handlePreview(file) {
-      console.log(file);
-    },
-    handleExceed(files, fileList) {
-      this.$message.warning(
-        `The limit is 2, you selected ${
-          files.length
-        } files this time, add up to ${files.length + fileList.length} totally`
-      );
-    },
-    handleChange(file,fileList){
-      this.fileList = fileList
-      this.filename = file.name
-      this.$emit('fileList',this.fileList);
-    }
-  },
-};
-</script>

frontend/src/components/upload/uploadCard.vue

@@ -1,68 +0,0 @@
-<template>
-  <el-card class="box-card">
-    <div slot="header" class="clearfix">
-      <span>Pose Upload</span>
-    </div>
-    <uploadForm @infor="infor"/>
-    <div id="upload-rules">
-      <p style="color:deepskyblue;font-weight: bold; font-size: 14px">Upload Rules</p>
-      <div v-for="item in rules" :key="item.index" class="rule">
-        <div>{{item.index}}. {{item.content}}</div>
-      </div>
-    </div>
-  </el-card>
-</template>
-
-<style>
-.text {
-  font-size: 14px;
-}
-
-.item {
-  margin-bottom: 18px;
-}
-
-.clearfix:after {
-  clear: both;
-}
-
-/* .box-card {
-  width: 480px;
-} */
-.box-card {
-  margin-left: 10%;
-  margin-right: 10%;
-  height: 520px;
-  /* text-align: center; */
-}
-#upload-rules{
-  position: relative;
-  text-align: left;
-  width: 500px;
-}
-</style>
-<script>
-import uploadForm from "./uploadForm.vue";
-
-export default {
-  name: "uploadCard",
-  data() {
-    return {
-      rules: [
-        {"index":1,"content":"Image must be png/jpg format."},
-        {"index":2,"content":"File name must be front/right.png."},
-        {"index":3,"content":"Height is measured in centimeter."},
-        {"index":4,"content":"Must upload 2 images (front and right) of only 1 person at a time."}
-      ],
-    };
-  },
-  components: {
-    uploadForm,
-  },
-  methods: {
-    infor(id) {
-      this.$emit("infor",id)
-    }
-  },
-};
-</script>

frontend/src/components/upload/uploadForm.vue

@@ -1,111 +0,0 @@
-<template>
-  <el-form ref="form" :model="form" label-width="120px">
-    <div id="form-content">
-      <el-form-item label="Pose" class="image-container">
-        <imageUpload @fileList="getFileList" />
-      </el-form-item>
-
-      <el-form-item label="Height" id="height">
-        <el-input
-          type="text"
-          placeholder="Please enter your height: (centimeter)"
-          autocomplete="off"
-          v-model="form.height"
-          size="small"
-          @input="getHeight"
-        ></el-input>
-      </el-form-item>
-    </div>
-    <el-form-item id="form-button-group">
-      <el-button
-        class="form-button"
-        type="primary"
-        size="small"
-        @click="onSubmit"
-        >Submit</el-button
-      >
-      <el-button class="form-button" disabled size="small">Cancel</el-button>
-    </el-form-item>
-    <el-dialog
-      title="提示"
-      :visible.sync="dialogVisible"
-      width="30%"
-      :before-close="handleClose"
-    >
-      <span>Your analysis ID is {{correction_id}}.</span>
-      <span slot="footer" class="dialog-footer">
-        <!-- <el-button @click="dialogVisible = false">Cancel</el-button> -->
-        <el-button type="primary" @click="dialogVisible = false"
-          >确认</el-button
-        >
-      </span>
-    </el-dialog>
-  </el-form>
-</template>
-<style scoped>
-#form-content {
-  border: 1px solid lightgrey;
-  padding: 10px 10px 10px 10px;
-  border-radius: 5px;
-}
-.image-container {
-  display: flex;
-  /* align-items: center; */
-  justify-content: center;
-  height: 6rem;
-}
-#form-button-group {
-  margin-top: 5%;
-  margin-left: 50%;
-  /* margin-right: -5em; */
-  /* padding-right: -10px; */
-}
-</style>
-<script>
-import imageUpload from "./imageUpload.vue";
-import axios from "axios";
-export default {
-  mounted() {},
-  data() {
-    return {
-      form: {
-        fileList: [],
-        poseList: [{ name: "front" }, { name: "side" }],
-        displayFlag: false,
-      },
-      dialogVisible: false,
-      correction_id: 0
-    };
-  },
-  components: {
-    imageUpload,
-  },
-  methods: {
-    getHeight(value) {
-      this.height = value;
-    },
-    getFileList(fileList) {
-      this.form.fileList = fileList;
-    },
-    onSubmit() {
-      // let _this = this;
-      let data = new FormData();
-      data.append("height", this.height);
-      Object.entries(this.form.fileList).forEach((file) => {
-        data.append("images" + file[1].name, file[1].raw);
-      });
-      axios
-        .post("http://106.15.1.178:8090/send_form", data)
-        .then((response) => {
-          this.correction_id = response.data.id;
-          // this.dialogVisible = true;
-          console.log(response.data.id)
-          this.$emit("infor", response.data.id);
-        })
-    },
-    handleClose(done) {
-      done();
-    },
-  },
-};
-</script>

frontend/src/main.js

@@ -2,11 +2,14 @@ import Vue from 'vue'
 import ElementUI from 'element-ui';
 import 'element-ui/lib/theme-chalk/index.css';
 import App from './App.vue';
+import router from './router'
+// import '@/src/util/request.js'
 
 Vue.use(ElementUI);
 
 Vue.config.productionTip = false
 
 new Vue({
-  render: h => h(App),
+  router,
+  render: h => h(App)
 }).$mount('#app')