OpsWise Automation Center for Hadoop

Hadoop Workflow Tools Survey  by Kevin Dempsey Peterson raises some interesting questions. Kevin suggests some key fuctions required of a Hadoop workflow/scheduler engine:-

Some I’m going to list what I think are the killer features to see in a Hadoop workflow scheduling system:

1. Schedule both map reduce jobs and other actions like copying a file from the local filesystem, or testing to ensure that a directory has 60 files in it.
2. Express a directed acyclic graph of dependencies between jobs and actions. (Loops would be nice, but I don’t need them.)
3. Full access to set arbitrary input formats, output formats, mapper, reducer, and combiner classes.
4. Ability to drop into Java code when needed with some sort of “postconfigure class”. I’m thinking of setting up a scanner for HBase TableInputFormat here.
5. Run as a server-side process. It should be possible for clients to submit entire workflows, and those workflows are then detached from the clients.
6. Ability to stop and restart a workflow part way through.
7. Ability to rerun a workflow which had a single part fail.
8. Ability to persist status between service restarts.
9. Scheduled jobs.

/* Style Definitions */ table.MsoNormalTable {mso-style-name:”Table Normal”; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:”"; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin-top:0cm; mso-para-margin-right:0cm; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0cm; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:”Calibri”,”sans-serif”; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:”Times New Roman”; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin;}

OpsWise Automation Center can do all of these functions and more! Like integrating Hadoop into z/OS batch runs!

KevinKevin Dempsey Peterson’s Blog kdpeterson.net/blog/2009/11/hadoop-workflow-tools-survey.html