lecture-21-openmp-work-sharing..md

Lecture 21: OpenMP Work Sharing.

Lecture Summary

• Last time: OpenMP nested parallelism, work sharing (for loops, sections)
• Today
  • OpenMP: nested parallelism, work sharing (tasks)
  • OpenMP: variable scoping, synchronization, loose ends

OpenMP Work Sharing

omp sections

Ending example

omp tasks

• Pros: Allows parallelization of irregular problems
  • Unbounded loops
  • Recursive algorithms
  • Producer/consumer
• Cons: Relatively tricky to deal with & introduce some overhead
• Motivations
  • OpenMP started to be tailored for large array-based applications
  • For example, the parallelization of a dynamic list traversal cannot be done in OpenMP for a long time
  • Storing pointers to list elements in an array: High overhead for array construction (not easy to parallelize)
  • Using single nowait inside a parallel region: High cost of the single construct. Also, each thread needs to traverse the entire list to determine if another thread has already processed that element
• Who does what and when?
  • The developer
    • Uses a pragma to specify where & what the tasks are
    • Ensures that there are no dependencies (that is, tasks can be executed independently)
  • The OpenMP runtime system
    • Generates a new task whenever a thread encounters a task construct
    • Decide the moment of execution (can be immediate or delayed)
• Definition: A task is a specific instance/combo of executable code along w/ its data environment (the shared & private data manipulated by the task) and ICV (internal control variables: thread scheduling and environment variables, typically associated with OpenMP)
• Synchronization issues. Solution: use task barriers (#pragma omp barrier, #pragma omp taskwait) to ensure the completion of tasks.

OpenMP Variable Scoping Issues

• Threads have access to a pool of memory that is shared
• Threads can also have private data
• Basic rule: Any variable declared prior to a parallel region is shared in that parallel region
• The private clause reproduces for each thread variables declared private in the pragma
• There are also OpenMP variables treated as private by default
  • Stack (local) variables in functions called from within parallel regions
  • Loop iteration variables
  • Automatic variables within a statement block
• When in doubt, always explicitly indicate something to be private
• firstprivate: Specifies that each thread should have its own instance of a variable. Moreover, the variable is initializes using the value of the variable of the same name from the master thread
  • Usage: #pragma omp parallel num_threads(4) firstprivate(i)
• lastprivate: The enclosing context's version of the variable is set equal to the private version of whichever thread executes the final iteration of the work-sharing construct (for or section)
• Data scoping is a common source of errors in OpenMP. It is the programmer's responsibility to make sure data dependencies do not lead to race conditions

OpenMP Synchronization

• Explicit barrier: #pragma omp barrier
• Implicit barriers: parallel, for, single, sections
• Unnecessary barriers hurt performance and can be removed with the nowait clause (applicable to for, single, sections)