cv_opticflow module

Compute optical flow.

Made for Linux video Devices. Computes x and y velocity using optical flow and distance to ground (using sonar).

Example for airframe file

Add to your firmware section:

<module name="cv_opticflow"/>

Module configuration options

Airframe file section

• section name: OPTICFLOW prefix: OPTICFLOW_
  • name AGL_ID value: ABI_BROADCAST
    Description: ABI receive id for ABI message (sonar measurement) (default: ABI_BROADCAST)
  • name SEND_ABI_ID value: 1
    Description: ABI sender id for opticflow messages (default: 1)
  • name CAMERA value: bottom_camera|front_camera
    Description: The V4L2 camera device that is used for the calculations
  • name DEROTATION_CORRECTION_FACTOR_X value: 1.0
    Description: Correction factor for derotation (in x direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)
  • name DEROTATION_CORRECTION_FACTOR_Y value: 1.0
    Description: Correction factor for derotation (in y direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)
  • name METHOD value: 0
    Description: Method used to calculate optical flow
  • name WINDOW_SIZE value: 10
    Description: Window size used for block matching (pixels)
  • name SEARCH_DISTANCE value: 10
    Description: Maximum search distance for blockmatching (pixels)
  • name SUBPIXEL_FACTOR value: 10
    Description: Amount of subpixels per pixel, used for more precise (subpixel) calculations of the flow
  • name RESOLUTION_FACTOR value: 1000
    Description: The resolution factor needed to calculate the divergence without floats
  • name DEROTATION value: 1
    Description: Derotation either turned on or off (depended on gyroscope measurements)
  • name MEDIAN_FILTER value: 0
    Description: A median filter on the resulting velocities to be turned on or off (last 5 measurements)
  • name FEATURE_MANAGEMENT value: 1
    Description: Whether to keep already tracked corners in memory for the next frame or re-detect new ones every time
  • name FPS value: 0
    Description: The (maximum) frequency to run the calculations at. If zero, it will max out at the camera frame rate
  • name TRACK_BACK value: TRUE
    Description: Whether flow vectors are tracked back to the previous image and only kept if they end up close to the original point.
  • name SHOW_FLOW value: TRUE
    Description: Whether to draw the flow vectors in the image.
  • name MAX_TRACK_CORNERS value: 25
    Description: The maximum amount of corners the Lucas Kanade algorithm is tracking between two frames
  • name MAX_ITERATIONS value: 10
    Description: Maximum number of iterations the Lucas Kanade algorithm should take
  • name THRESHOLD_VEC value: 2
    Description: TThreshold in subpixels when the iterations of Lucas Kanade should stop
  • name CORNER_METHOD value: 1
    Description: Method used to look for corners, exhaustive FAST (0) or ACT-FAST (1).
  • name FAST9_ADAPTIVE value: TRUE
    Description: Whether we should use and adapative FAST9 crner detection threshold
  • name FAST9_THRESHOLD value: 20
    Description: FAST9 default threshold
  • name FAST9_MIN_DISTANCE value: 10
    Description: The amount of pixels between corners that should be detected
  • name FAST9_PADDING value: 20
    Description: The outer border in which no corners will be searched
  • name FAST9_REGION_DETECT value: 1
    Description: Whether to detect fast9 corners in regions of interest or the whole image (only works with feature management)
  • name FAST9_NUM_REGIONS value: 9
    Description: The number of regions of interest to split the image into
  • name ACTFAST_LONG_STEP value: 10
    Description: Step size to take when there is no texture
  • name ACTFAST_SHORT_STEP value: 2
    Description: Step size to take when there is an edge to be followed
  • name ACTFAST_MIN_GRADIENT value: 10
    Description: Threshold that decides when there is sufficient texture for edge following
  • name ACTFAST_GRADIENT_METHOD value: 1
    Description: Whether to use a simple (0) or Sobel (1) filter

Auto-loaded modules

The following modules are automatically loaded (just as if you had added them in the airframe file)

• agl_dist module
• video_thread module
• pose_history module

Module functions

Init Functions

These initialization functions are called once on startup.

• opticflow_module_init()

Periodic Functions

These functions are called periodically at the specified frequency from the module periodic loop.

• opticflow_module_run()
  • Running at maximum module frequency.
  • Autorun: TRUE
    Periodic function automatically starts after init.

Files

Header Files

The following headers are automatically included in modules.h

• opticflow_module.h

Source Files

• modules/computer_vision/lib/vision/image.c
• modules/computer_vision/lib/encoding/jpeg.c
• modules/computer_vision/lib/encoding/rtp.c
• modules/computer_vision/opticflow_module.c
• modules/computer_vision/opticflow/opticflow_calculator.c
• modules/computer_vision/opticflow/size_divergence.c
• modules/computer_vision/opticflow/linear_flow_fit.c
• math/pprz_algebra_float.c
• math/pprz_matrix_decomp_float.c
• modules/computer_vision/lib/vision/act_fast.c
• modules/computer_vision/lib/vision/fast_rosten.c
• modules/computer_vision/lib/vision/lucas_kanade.c
• modules/computer_vision/lib/vision/edge_flow.c
• modules/computer_vision/lib/vision/undistortion.c

Raw cv_opticflow.xml file:

<!DOCTYPE module SYSTEM "module.dtd">

<module name="cv_opticflow" dir="computer_vision">
  <doc>
    <description>
      Compute optical flow.
      Made for Linux video Devices.
      Computes x and y velocity using optical flow and distance to ground (using sonar).
    </description>

    <!-- Optical flow calculation parameters -->
    <section name="OPTICFLOW" prefix="OPTICFLOW_">
      <define name="AGL_ID" value="ABI_BROADCAST" description="ABI receive id for ABI message (sonar measurement) (default: ABI_BROADCAST)"/>
      <define name="SEND_ABI_ID" value="1" description="ABI sender id for opticflow messages (default: 1)"/>

      <!-- Video device parameters -->
      <define name="CAMERA" value="bottom_camera|front_camera" description="The V4L2 camera device that is used for the calculations"/>

      <!-- Camera parameters -->
      <define name="DEROTATION_CORRECTION_FACTOR_X" value="1.0" description="Correction factor for derotation (in x direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)"/>
      <define name="DEROTATION_CORRECTION_FACTOR_Y" value="1.0" description="Correction factor for derotation (in y direction), estimated from a fit between the gyro's rates and the resulting flow (caused by the camera not being exactly in the middle (Defaults are from an ARDrone 2)"/>
      <configure name="BODY_TO_CAM_PHI" value="0" description="Rotation from body frame to camera frame around x axis"/>
      <configure name="BODY_TO_CAM_THETA" value="0" description="Rotation from body frame to camera frame around y axis"/>
      <configure name="BODY_TO_CAM_PSI" value="0" description="Rotation from body frame to camera frame around z axis"/>

      <!-- General optical flow calculation parameters -->
      <define name="METHOD" value="0" description="Method used to calculate optical flow"/>
      <define name="WINDOW_SIZE" value="10" description="Window size used for block matching (pixels)"/>
      <define name="SEARCH_DISTANCE" value="10" description="Maximum search distance for blockmatching (pixels)"/>
      <define name="SUBPIXEL_FACTOR" value="10" description="Amount of subpixels per pixel, used for more precise (subpixel) calculations of the flow"/>
      <define name="RESOLUTION_FACTOR" value="1000" description="The resolution factor needed to calculate the divergence without floats"/>
      <define name="DEROTATION" value="1" description="Derotation either turned on or off (depended on gyroscope measurements)"/>
      <define name="MEDIAN_FILTER" value="0" description="A median filter on the resulting velocities to be turned on or off (last 5 measurements)"/>
      <define name="FEATURE_MANAGEMENT" value="1" description="Whether to keep already tracked corners in memory for the next frame or re-detect new ones every time"/>
      <define name="FPS" value="0" description="The (maximum) frequency to run the calculations at. If zero, it will max out at the camera frame rate"/>
      <define name="TRACK_BACK" value="TRUE" description="Whether flow vectors are tracked back to the previous image and only kept if they end up close to the original point."/>
      <define name="SHOW_FLOW" value="TRUE" description="Whether to draw the flow vectors in the image."/>

      <!-- Lucas Kanade optical flow calculation parameters -->
      <define name="MAX_TRACK_CORNERS" value="25" description="The maximum amount of corners the Lucas Kanade algorithm is tracking between two frames"/>
      <define name="MAX_ITERATIONS" value="10" description="Maximum number of iterations the Lucas Kanade algorithm should take"/>
      <define name="THRESHOLD_VEC" value="2" description="TThreshold in subpixels when the iterations of Lucas Kanade should stop"/>

      <define name="CORNER_METHOD" value="1" description="Method used to look for corners, exhaustive FAST (0) or ACT-FAST (1)."/>

      <!-- FAST9 corner detection parameters -->
      <define name="FAST9_ADAPTIVE" value="TRUE" description="Whether we should use and adapative FAST9 crner detection threshold"/>
      <define name="FAST9_THRESHOLD" value="20" description="FAST9 default threshold"/>
      <define name="FAST9_MIN_DISTANCE" value="10" description="The amount of pixels between corners that should be detected"/>
      <define name="FAST9_PADDING" value="20" description="The outer border in which no corners will be searched"/>
      <define name="FAST9_REGION_DETECT" value="1" description="Whether to detect fast9 corners in regions of interest or the whole image (only works with feature management)"/>
      <define name="FAST9_NUM_REGIONS" value="9" description="The number of regions of interest to split the image into"/>

      <!-- ACT-FAST parameters -->
      <define name="ACTFAST_LONG_STEP" value="10" description="Step size to take when there is no texture"/>
      <define name="ACTFAST_SHORT_STEP" value="2" description="Step size to take when there is an edge to be followed"/>
      <define name="ACTFAST_MIN_GRADIENT" value="10" description="Threshold that decides when there is sufficient texture for edge following"/>
      <define name="ACTFAST_GRADIENT_METHOD" value="1" description="Whether to use a simple (0) or Sobel (1) filter"/>

    </section>
  </doc>

  <settings>
  <dl_settings NAME="Vision stabilization">
      <!-- Optical flow calculations parameters -->
      <dl_settings name="vision_calc">
      <dl_setting var="opticflow.method" min="0" step="1" max="1" module="computer_vision/opticflow_module" shortname="method" values="LK_Fast9|EdgeFlow" param="METHOD"/>
      <dl_setting var="opticflow.corner_method" min="0" step="1" max="1" module="computer_vision/opticflow_module" shortname="corner_method" values="exhaustive-FAST|ACT-FAST" param="CORNER_METHOD"/>
        <dl_setting var="opticflow.window_size" module="computer_vision/opticflow_module" min="0" step="1" max="20" shortname="window_size" param="OPTICFLOW_WINDOW_SIZE"/>
        <dl_setting var="opticflow.search_distance" module="computer_vision/opticflow_module" min="0" step="1" max="50" shortname="search_distance" param="SEARCH_DISTANCE"/>
        <dl_setting var="opticflow.subpixel_factor" module="computer_vision/opticflow_module" min="0" step="10" max="1000" shortname="subpixel_factor" param="OPTICFLOW_SUBPIXEL_FACTOR"/>
        <dl_setting var="opticflow.resolution_factor" module="computer_vision/opticflow_module" min="10" step="10" max="10000" shortname="resolution_factor" param="OPTICFLOW_RESOLUTION_FACTOR"/>
        <dl_setting var="opticflow.derotation" min="0" step="1" max="1" module="computer_vision/opticflow_module" values="OFF|ON" shortname="derotation" param="OPTICFLOW_DEROTATION"/>
        <dl_setting var="opticflow.median_filter" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="median_filter" param="OPTICFLOW_MEDIAN_FILTER"/>
        <dl_setting var="opticflow.feature_management" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="feature_management" param="OPTICFLOW_FEATURE_MANAGEMENT"/>
        <dl_setting var="opticflow.track_back" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="FALSE|TRUE" shortname="track_back" param="OPTICFLOW_TRACK_BACK"/>
        <dl_setting var="opticflow.show_flow" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="FALSE|TRUE" shortname="show_flow" param="OPTICFLOW_SHOW_FLOW"/>


        <!-- Specifically for Lucas Kanade and FAST9 -->
        <dl_setting var="opticflow.max_track_corners" module="computer_vision/opticflow_module" min="0" step="1" max="500" shortname="max_trck_corners" param="OPTICFLOW_MAX_TRACK_CORNERS"/>
        <dl_setting var="opticflow.max_iterations" module="computer_vision/opticflow_module" min="0" step="1" max="100" shortname="max_iterations" param="OPTICFLOW_MAX_ITERATIONS"/>
        <dl_setting var="opticflow.threshold_vec" module="computer_vision/opticflow_module" min="0" step="1" max="100" shortname="threshold_vec" param="OPTICFLOW_THRESHOLD_VEC"/>
        <dl_setting var="opticflow.fast9_adaptive" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="FALSE|TRUE" shortname="fast9_adaptive" param="OPTICFLOW_FAST9_ADAPTIVE"/>
        <dl_setting var="opticflow.fast9_threshold" module="computer_vision/opticflow_module" min="0" step="1" max="255" shortname="fast9_threshold" param="OPTICFLOW_FAST9_THRESHOLD"/>
        <dl_setting var="opticflow.fast9_min_distance" module="computer_vision/opticflow_module" min="0" step="1" max="500" shortname="fast9_min_distance" param="OPTICFLOW_FAST9_MIN_DISTANCE"/>
        <dl_setting var="opticflow.fast9_padding" module="computer_vision/opticflow_module" min="0" step="1" max="50" shortname="fast9_padding" param="OPTICFLOW_FAST9_PADDING"/>
        <dl_setting var="opticflow.fast9_region_detect" module="computer_vision/opticflow_module" min="0" step="1" max="1" values="OFF|ON" shortname="fast9_region_detect" param="OPTICFLOW_FAST9_REGION_DETECT"/>
        <dl_setting var="opticflow.fast9_num_regions" module="computer_vision/opticflow_module" min="1" step="1" max="25" shortname="fast9_num_regions" param="OPTICFLOW_FAST9_NUM_REGIONS"/>

        <!-- ACT-FAST settings -->
        <dl_setting var="opticflow.actfast_long_step" module="computer_vision/opticflow_module" min="1" step="1" max="100" shortname="actfast_long_step" param="OPTICFLOW_ACTFAST_LONG_STEP"/>
        <dl_setting var="opticflow.actfast_short_step" module="computer_vision/opticflow_module" min="1" step="1" max="10" shortname="actfast_short_step" param="OPTICFLOW_ACTFAST_SHORT_STEP"/>
        <dl_setting var="opticflow.actfast_min_gradient" module="computer_vision/opticflow_module" min="1" step="1" max="255" shortname="actfast_min_gradient" param="OPTICFLOW_ACTFAST_MIN_GRADIENT"/>
        <dl_setting var="opticflow.actfast_gradient_method" module="computer_vision/opticflow_module" min="0" step="1" max="1"  values="SIMPLE|SOBEL" shortname="actfast_gradient_method" param="OPTICFLOW_ACTFAST_GRADIENT_METHOD"/>        

        <!-- Changes pyramid level of lucas kanade optical flow. -->
        <dl_setting var="opticflow.pyramid_level" module="computer_vision/opticflow_module" min="0" step="1" max="10" shortname="pyramid_level" param="OPTICFLOW_PYRAMID_LEVEL"/>
      </dl_settings>
    </dl_settings>
  </settings>

  <autoload name="agl_dist"/>
  <autoload name="video_thread"/>
  <autoload name="pose_history"/>

  <header>
    <file name="opticflow_module.h"/>
  </header>

  <init fun="opticflow_module_init()"/>
  <periodic fun="opticflow_module_run()" autorun="TRUE"/>

  <makefile target="ap|nps">
    <!-- Include the needed Computer Vision files -->
    <include name="modules/computer_vision"/>
    <file name="image.c" dir="modules/computer_vision/lib/vision"/>
    <file name="jpeg.c" dir="modules/computer_vision/lib/encoding"/>
    <file name="rtp.c" dir="modules/computer_vision/lib/encoding"/>

    <!-- The optical flow module (calculator) -->
    <file name="opticflow_module.c"/>
    <file name="opticflow_calculator.c" dir="modules/computer_vision/opticflow"/>
    <file name="size_divergence.c" dir="modules/computer_vision/opticflow"/>
    <file name="linear_flow_fit.c" dir="modules/computer_vision/opticflow"/>
    <file name="pprz_algebra_float.c" dir="math"/>
    <file name="pprz_matrix_decomp_float.c" dir="math"/>

    <!-- Main vision calculations -->
    <file name="act_fast.c" dir="modules/computer_vision/lib/vision"/>
    <file name="fast_rosten.c" dir="modules/computer_vision/lib/vision"/>
    <file name="lucas_kanade.c" dir="modules/computer_vision/lib/vision"/>
    <file name="edge_flow.c" dir="modules/computer_vision/lib/vision"/>
    <file name="undistortion.c" dir="modules/computer_vision/lib/vision"/>
  </makefile>
</module>