cv_opticflow module

Compute optical flow.

Made for Linux video Devices. Computes x and y velocity using optical flow and distance to ground (using sonar).

Example for airframe file

Add to your firmware section:

<module name="video_thread"/>
<module name="pose_history"/>
<module name="cv_opticflow"/>

Module configuration options

Airframe file section

• section name: OPTICFLOW prefix: OPTICFLOW_
  • name AGL_ID value: ABI_BROADCAST
    Description: ABI receive id for ABI message (sonar measurement) (default: ABI_BROADCAST)
  • name IMU_ID value: ABI_BROADCAST
    Description: ABI receive id for ABI message (accelerometers) (default: ABI_BROADCAST)
  • name BODY_TO_IMU_ID value: ABI_BROADCAST
    Description: ABI receive id for ABI message (body to imu quat) (default: ABI_BROADCAST)
  • name SEND_ABI_ID value: 1
    Description: ABI sender id for opticflow messages (default: 1)
  • name CAMERA value: bottom_camera|front_camera
    Description: The V4L2 camera device that is used for the calculations
  • name FOV_W value: 0.89360857702
    Description: The field of view width of the bottom camera (Defaults are from an ARDrone 2)
  • name FOV_H value: 0.67020643276
    Description: The field of view height of the bottom camera (Defaults are from an ARDrone 2)
  • name FX value: 343.1211
    Description: Field in the x direction of the camera (Defaults are from an ARDrone 2)
  • name FY value: 348.5053
    Description: Field in the y direction of the camera (Defaults are from an ARDrone 2)
  • name DEROTATION_CORRECTION_FACTOR_X value: 1.0
    Description: Correction factor for derotation (in x direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)
  • name DEROTATION_CORRECTION_FACTOR_Y value: 1.0
    Description: Correction factor for derotation (in y direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)
  • name METHOD value: 0
    Description: Method used to calculate optical flow
  • name WINDOW_SIZE value: 10
    Description: Window size used for block matching (pixels)
  • name SEARCH_DISTANCE value: 10
    Description: Maximum search distance for blockmatching (pixels)
  • name SUBPIXEL_FACTOR value: 10
    Description: Amount of subpixels per pixel, used for more precise (subpixel) calculations of the flow
  • name RESOLUTION_FACTOR value: 1000
    Description: The resolution factor needed to calculate the divergence without floats
  • name DEROTATION value: 1
    Description: Derotation either turned on or off (depended on gyroscope measurements)
  • name MEDIAN_FILTER value: 0
    Description: A median filter on the resulting velocities to be turned on or off (last 5 measurements)
  • name KALMAN_FILTER value: 1
    Description: A kalman filter on the resulting velocities to be turned on or off (fused with accelerometers)
  • name KALMAN_FILTER_PROCESS_NOISE value: 0.01
    Description: The expected variance of the error of the model's prediction in the kalman filter
  • name FEATURE_MANAGEMENT value: 1
    Description: Whether to keep already tracked corners in memory for the next frame or re-detect new ones every time
  • name FPS value: 0
    Description: The (maximum) frequency to run the calculations at. If zero, it will max out at the camera frame rate
  • name MAX_TRACK_CORNERS value: 25
    Description: The maximum amount of corners the Lucas Kanade algorithm is tracking between two frames
  • name MAX_ITERATIONS value: 10
    Description: Maximum number of iterations the Lucas Kanade algorithm should take
  • name THRESHOLD_VEC value: 2
    Description: TThreshold in subpixels when the iterations of Lucas Kanade should stop
  • name FAST9_ADAPTIVE value: TRUE
    Description: Whether we should use and adapative FAST9 crner detection threshold
  • name FAST9_THRESHOLD value: 20
    Description: FAST9 default threshold
  • name FAST9_MIN_DISTANCE value: 10
    Description: The amount of pixels between corners that should be detected
  • name FAST9_PADDING value: 20
    Description: The outer border in which no corners will be searched
  • name FAST9_REGION_DETECT value: 1
    Description: Whether to detect fast9 corners in regions of interest or the whole image (only works with feature management)
  • name FAST9_NUM_REGIONS value: 9
    Description: The number of regions of interest to split the image into

Dependencies

• video_thread module
• pose_history module

Module functions

Init Functions

These initialization functions are called once on startup.

• opticflow_module_init()

Periodic Functions

These functions are called periodically at the specified frequency from the module periodic loop.

• opticflow_module_run()
  • Running at maximum module frequency.
  • Autorun: TRUE
    Periodic function automatically starts after init.

Files

Header Files

The following headers are automatically included in modules.h

• opticflow_module.h

Source Files

• modules/computer_vision/lib/vision/image.c
• modules/computer_vision/lib/encoding/jpeg.c
• modules/computer_vision/lib/encoding/rtp.c
• modules/computer_vision/lib/v4l/v4l2.c
• modules/computer_vision/opticflow_module.c
• modules/computer_vision/opticflow/opticflow_calculator.c
• modules/computer_vision/opticflow/size_divergence.c
• modules/computer_vision/opticflow/linear_flow_fit.c
• math/pprz_algebra_float.c
• math/pprz_matrix_decomp_float.c
• modules/computer_vision/lib/vision/fast_rosten.c
• modules/computer_vision/lib/vision/lucas_kanade.c
• modules/computer_vision/lib/vision/edge_flow.c
• modules/computer_vision/lib/filters/kalman_filter_vision.c

Raw cv_opticflow.xml file:

<!DOCTYPE module SYSTEM "module.dtd">

<module name="cv_opticflow" dir="computer_vision">
  <doc>
    <description>
      Compute optical flow.
      Made for Linux video Devices.
      Computes x and y velocity using optical flow and distance to ground (using sonar).
    </description>

    <!-- Optical flow calculation parameters -->
    <section name="OPTICFLOW" prefix="OPTICFLOW_">
      <define name="AGL_ID" value="ABI_BROADCAST" description="ABI receive id for ABI message (sonar measurement) (default: ABI_BROADCAST)"/>
<define name="IMU_ID" value="ABI_BROADCAST" description="ABI receive id for ABI message (accelerometers) (default: ABI_BROADCAST)"/>
<define name="BODY_TO_IMU_ID" value="ABI_BROADCAST" description="ABI receive id for ABI message (body to imu quat) (default: ABI_BROADCAST)"/>
      <define name="SEND_ABI_ID" value="1" description="ABI sender id for opticflow messages (default: 1)"/>

      <!-- Video device parameters -->
      <define name="CAMERA" value="bottom_camera|front_camera" description="The V4L2 camera device that is used for the calculations"/>

      <!-- Camera parameters -->
      <define name="FOV_W" value="0.89360857702" description="The field of view width of the bottom camera (Defaults are from an ARDrone 2)"/>
      <define name="FOV_H" value="0.67020643276" description="The field of view height of the bottom camera (Defaults are from an ARDrone 2)"/>
      <define name="FX" value="343.1211" description="Field in the x direction of the camera (Defaults are from an ARDrone 2)"/>
      <define name="FY" value="348.5053" description="Field in the y direction of the camera (Defaults are from an ARDrone 2)"/>
      <define name="DEROTATION_CORRECTION_FACTOR_X" value="1.0" description="Correction factor for derotation (in x direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)"/>
      <define name="DEROTATION_CORRECTION_FACTOR_Y" value="1.0" description="Correction factor for derotation (in y direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)"/>

      <!-- General optical flow calculation parameters -->
      <define name="METHOD" value="0" description="Method used to calculate optical flow"/>
      <define name="WINDOW_SIZE" value="10" description="Window size used for block matching (pixels)"/>
      <define name="SEARCH_DISTANCE" value="10" description="Maximum search distance for blockmatching (pixels)"/>
      <define name="SUBPIXEL_FACTOR" value="10" description="Amount of subpixels per pixel, used for more precise (subpixel) calculations of the flow"/>
      <define name="RESOLUTION_FACTOR" value="1000" description="The resolution factor needed to calculate the divergence without floats"/>
      <define name="DEROTATION" value="1" description="Derotation either turned on or off (depended on gyroscope measurements)"/>
      <define name="MEDIAN_FILTER" value="0" description="A median filter on the resulting velocities to be turned on or off (last 5 measurements)"/>
      <define name="KALMAN_FILTER" value="1" description="A kalman filter on the resulting velocities to be turned on or off (fused with accelerometers)"/>
      <define name="KALMAN_FILTER_PROCESS_NOISE" value="0.01" description="The expected variance of the error of the model's prediction in the kalman filter"/>
      <define name="FEATURE_MANAGEMENT" value="1" description="Whether to keep already tracked corners in memory for the next frame or re-detect new ones every time"/>
      <define name="FPS" value="0" description="The (maximum) frequency to run the calculations at. If zero, it will max out at the camera frame rate"/>

      <!-- Lucas Kanade optical flow calculation parameters -->
      <define name="MAX_TRACK_CORNERS" value="25" description="The maximum amount of corners the Lucas Kanade algorithm is tracking between two frames"/>
      <define name="MAX_ITERATIONS" value="10" description="Maximum number of iterations the Lucas Kanade algorithm should take"/>
      <define name="THRESHOLD_VEC" value="2" description="TThreshold in subpixels when the iterations of Lucas Kanade should stop"/>

      <!-- FAST9 corner detection parameters -->
      <define name="FAST9_ADAPTIVE" value="TRUE" description="Whether we should use and adapative FAST9 crner detection threshold"/>
      <define name="FAST9_THRESHOLD" value="20" description="FAST9 default threshold"/>
      <define name="FAST9_MIN_DISTANCE" value="10" description="The amount of pixels between corners that should be detected"/>
      <define name="FAST9_PADDING" value="20" description="The outer border in which no corners will be searched"/>
      <define name="FAST9_REGION_DETECT" value="1" description="Whether to detect fast9 corners in regions of interest or the whole image (only works with feature management)"/>
      <define name="FAST9_NUM_REGIONS" value="9" description="The number of regions of interest to split the image into"/>
    </section>
  </doc>

  <settings>
  <dl_settings NAME="Vision stabilization">

        <!-- Optical flow calculations parameters -->
      <dl_settings name="vision_calc">
      <dl_setting var="opticflow.method" min="0" step="1" max="1" module="computer_vision/opticflow_module" shortname="method" values="LK_Fast9|EdgeFlow" param="METHOD"/>
        <dl_setting var="opticflow.window_size" module="computer_vision/opticflow_module" min="0" step="1" max="20" shortname="window_size" param="OPTICFLOW_WINDOW_SIZE"/>
        <dl_setting var="opticflow.search_distance" module="computer_vision/opticflow_module" min="0" step="1" max="50" shortname="search_distance" param="SEARCH_DISTANCE"/>
        <dl_setting var="opticflow.subpixel_factor" module="computer_vision/opticflow_module" min="0" step="10" max="1000" shortname="subpixel_factor" param="OPTICFLOW_SUBPIXEL_FACTOR"/>
        <dl_setting var="opticflow.resolution_factor" module="computer_vision/opticflow_module" min="10" step="10" max="10000" shortname="resolution_factor" param="OPTICFLOW_RESOLUTION_FACTOR"/>
        <dl_setting var="opticflow.derotation" min="0" step="1" max="1" module="computer_vision/opticflow_module" values="OFF|ON" shortname="derotation" param="OPTICFLOW_DEROTATION"/>
        <dl_setting var="opticflow.median_filter" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="median_filter" param="OPTICFLOW_MEDIAN_FILTER"/>
        <dl_setting var="opticflow.kalman_filter" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="kalman_filter" param="OPTICFLOW_KALMAN_FILTER"/>
        <dl_setting var="opticflow.kalman_filter_process_noise" module="computer_vision/opticflow_module" min="0.0001" step="0.0001" max="0.1"  shortname="KF_process_noise" param="OPTICFLOW_KALMAN_FILTER_PROCESS_NOISE"/>
        <dl_setting var="opticflow.feature_management" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="feature_management" param="OPTICFLOW_FEATURE_MANAGEMENT"/>

  <!-- Specifically for Lucas Kanade and FAST9 -->
        <dl_setting var="opticflow.max_track_corners" module="computer_vision/opticflow_module" min="0" step="1" max="500" shortname="max_trck_corners" param="OPTICFLOW_MAX_TRACK_CORNERS"/>
        <dl_setting var="opticflow.max_iterations" module="computer_vision/opticflow_module" min="0" step="1" max="100" shortname="max_iterations" param="OPTICFLOW_MAX_ITERATIONS"/>
        <dl_setting var="opticflow.threshold_vec" module="computer_vision/opticflow_module" min="0" step="1" max="100" shortname="threshold_vec" param="OPTICFLOW_THRESHOLD_VEC"/>

        <dl_setting var="opticflow.fast9_adaptive" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="FALSE|TRUE" shortname="fast9_adaptive" param="OPTICFLOW_FAST9_ADAPTIVE"/>
        <dl_setting var="opticflow.fast9_threshold" module="computer_vision/opticflow_module" min="0" step="1" max="255" shortname="fast9_threshold" param="OPTICFLOW_FAST9_THRESHOLD"/>
        <dl_setting var="opticflow.fast9_min_distance" module="computer_vision/opticflow_module" min="0" step="1" max="500" shortname="fast9_min_distance" param="OPTICFLOW_FAST9_MIN_DISTANCE"/>
        <dl_setting var="opticflow.fast9_padding" module="computer_vision/opticflow_module" min="0" step="1" max="50" shortname="fast9_padding" param="OPTICFLOW_FAST9_PADDING"/>
        <dl_setting var="opticflow.fast9_region_detect" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="fast9_region_detect" param="OPTICFLOW_FAST9_REGION_DETECT"/>
        <dl_setting var="opticflow.fast9_num_regions" module="computer_vision/opticflow_module" min="1" step="1" max="25" shortname="fast9_num_regions" param="OPTICFLOW_FAST9_NUM_REGIONS"/>


  <!-- Changes pyramid level of lucas kanade optical flow. -->
        <dl_setting var="opticflow.pyramid_level" module="computer_vision/opticflow_module" min="0" step="1" max="10" shortname="pyramid_level" param="OPTICFLOW_PYRAMID_LEVEL"/>

      </dl_settings>
    </dl_settings>
  </settings>

  <depends>video_thread,pose_history</depends>

  <header>
    <file name="opticflow_module.h"/>
  </header>

  <init fun="opticflow_module_init()"/>
  <periodic fun="opticflow_module_run()" autorun="TRUE"/>

  <makefile target="ap">
    <!-- Include the needed Computer Vision files -->
    <include name="modules/computer_vision"/>
    <file name="image.c" dir="modules/computer_vision/lib/vision"/>
    <file name="jpeg.c" dir="modules/computer_vision/lib/encoding"/>
    <file name="rtp.c" dir="modules/computer_vision/lib/encoding"/>
    <file name="v4l2.c" dir="modules/computer_vision/lib/v4l"/>

    <!-- The optical flow module (calculator) -->
    <file name="opticflow_module.c"/>
    <file name="opticflow_calculator.c" dir="modules/computer_vision/opticflow"/>
    <file name="size_divergence.c" dir="modules/computer_vision/opticflow"/>
    <file name="linear_flow_fit.c" dir="modules/computer_vision/opticflow"/>
    <file name="pprz_algebra_float.c" dir="math"/>
    <file name="pprz_matrix_decomp_float.c" dir="math"/>

    <!-- Main vision calculations -->
    <file name="fast_rosten.c" dir="modules/computer_vision/lib/vision"/>
    <file name="lucas_kanade.c" dir="modules/computer_vision/lib/vision"/>
    <file name="edge_flow.c" dir="modules/computer_vision/lib/vision"/>
    <file name="kalman_filter_vision.c" dir="modules/computer_vision/lib/filters"/>
  </makefile>


</module>