Fri
07
Jan 2011

To run a console process programatically from your program and capture its output is very difficult to do right. There are so many possible problems! Not only you have to pass correct command line parameters and escape them if they contain special characters like space, care about working directory and maybe even environmental variables. You must also handle different possible behavior of the program. For example, it can freeze indefinitely, so you better do some timeout when waiting for it to exit. It may ask for additional console input and wait for it. It may print some information on both of its outputs - "standard output" and "standard error". They are all asynchronous and you better capture these data because if a buffer fills up, the program will hang waiting until you consume this output to allow it to post more. You also have to read these buffers to the end after program exited because there may still be some bytes remaining.

I do it in C# and I was hoping I already did it all right... Until today. I wanted to run "mysqldump" and capture its output to store a MySQL database table in memory and then import it on another machine. But when exporting a table with large binary data in its BLOB fields, my program hung. My code was like this:

using System.Text;
using System.Diagnostics;
using System.Threading;

class ConsoleCommand
{
  public void Execute(string fileName, string arguments, int timeout, string standardInputData)
  {
    using (Process process = new Process())
    {
      process.StartInfo.FileName = fileName;
      process.StartInfo.Arguments = arguments;
      process.StartInfo.RedirectStandardInput = true;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = true;
      process.OutputDataReceived += new DataReceivedEventHandler(StandardOutputHandler);
      process.ErrorDataReceived += new DataReceivedEventHandler(StandardErrorHandler);
      process.Start();
  
      process.BeginOutputReadLine();
      process.BeginErrorReadLine();
  
      if (standardInputData != null)
      {
        process.StandardInput.Write(standardInputData);
        process.StandardInput.Close();
      }
  
      if (!process.WaitForExit(timeout))
        throw new Exception("Process execution timeout.");
      m_StandardOutputEvent.WaitOne();
      m_StandardErrorEvent.WaitOne();
  
      m_ExitCode = process.ExitCode;
    }
  }

  public int ExitCode { get { return m_ExitCode; } }
  public string StandardOutputData { get { return m_StandardOutputData.ToString(); } }
  public string StandardErrorData { get { return m_StandardErrorData.ToString(); } }

  private void StandardOutputHandler(object process, DataReceivedEventArgs e)
  {
    if (e.Data == null)
      m_StandardOutputEvent.Set();
    else
      m_StandardOutputData.AppendLine(e.Data);
  }

  private void StandardErrorHandler(object process, DataReceivedEventArgs e)
  {
    if (e.Data == null)
      m_StandardErrorEvent.Set();
    else
      m_StandardErrorData.AppendLine(e.Data);
  }

  private int m_ExitCode;
  private StringBuilder m_StandardOutputData = new StringBuilder();
  private StringBuilder m_StandardErrorData = new StringBuilder();
  private ManualResetEvent m_StandardOutputEvent = new ManualResetEvent(false);
  private ManualResetEvent m_StandardErrorEvent = new ManualResetEvent(false);
}

I couldn't find any solution on the Internet. That's why I write about it here - maybe some day someone (you?) who has same problem finds this article on Google. I know it must have something in common with binary data on the output, so I coded a small program that printed all possible ASCII characters. It worked. So there was not the problem with characters codes, but with the length of the data. The delegate associated with OutputDataReceived and ErrorDataReceived is designed to handle text lines. I suppose there is an internal buffer somewhere inside .NET that splits the text into lines. If there is a lot of data (more than 1 MB) in a single line (binary data without end-of-line character), the buffer fills up, neither OutputDataReceived delegate is called, nor the console program is able to print more output and everything just deadlocks :(

The only solution I know at the moment is not to use these OutputDataReceived and ErrorDataReceived events, but to poll for output data synchronously like this:

class ConsoleCommand
{
  public void Execute(string fileName, string arguments, int timeout, string standardInputData)
  {
    using (Process process = new Process())
    {
      process.StartInfo.FileName = fileName;
      process.StartInfo.Arguments = arguments;
      process.StartInfo.RedirectStandardInput = true;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = true;
      process.Start();

      if (standardInputData != null)
      {
        process.StandardInput.Write(standardInputData);
        process.StandardInput.Close();
      }

      DateTime end_time = DateTime.Now + TimeSpan.FromMilliseconds((double)timeout);
      bool process_exit = false, output_end = false, error_end = false;
      for (;;)
      {
        if (!output_end)
        {
          if (process.StandardOutput.EndOfStream)
            output_end = true;
          else
            m_StandardOutputData.Append(process.StandardOutput.ReadToEnd());
        }

        if (!error_end)
        {
          if (process.StandardError.EndOfStream)
            error_end = true;
          else
            m_StandardErrorData.Append(process.StandardError.ReadToEnd());
        }

        if (!process_exit)
          process_exit = process.WaitForExit(0);

        if (DateTime.Now > end_time)
        {
          process.Close();
          throw new Exception("Process execution timeout.");
        }

        if (process_exit && output_end && error_end)
          break;

        System.Threading.Thread.Yield();
      }

      m_ExitCode = process.ExitCode;
    }
  }

  public int ExitCode { get { return m_ExitCode; } }
  public string StandardOutputData { get { return m_StandardOutputData.ToString(); } }
  public string StandardErrorData { get { return m_StandardErrorData.ToString(); } }

  private int m_ExitCode;
  private StringBuilder m_StandardOutputData = new StringBuilder();
  private StringBuilder m_StandardErrorData = new StringBuilder();
}

Comments | #.net Share

Comments